How Do Shopee Engineers Use Big Data to Create A Big Impact?

What role does big data engineering play in building a leading e-commerce platform that impacts millions of users across the globe? We speak to our Shopee data engineers to find out how they acquire and utilise big data to serve business teams across the organisation, and create a better shopping experience for our users!

Tell us about yourself! What are you passionate about?

Ravi (R): Hi, I’m Ravi, a data engineer at Shopee. I’ve always loved chess since I was a child – this game never fails to amaze me. It reminds me that in both life and chess, every step we take has consequences and we should try to make the best choices to develop positive outcomes for ourselves. I’m also learning Mandarin through a programme designed by the Shopee Learning and Development team – and I like that learning the language helps me to appreciate Chinese traditions, art, and history better.

Beini (B): Hey everyone, I’m Beini, and I’m also a Shopee data engineer. At the moment, I like spending my leisure time playing with my jazz drum set. I’m in the process of learning to play the song ‘Yellow’ by Coldplay, which is one of my favourite songs. It’s a really great activity to help me unwind!

What got you into the Data Engineering field?

Ravi (R): When I was in college, I became fascinated with how software on e-commerce platforms could recognise my browsing patterns and recommend suitable products for me. Fast-forward to a few years of working in the industry, I gradually learned how artificial intelligence and machine learning can be utilised to build these search and recommendation systems. A large set of data is first required for our data scientists to conduct analysis and build these advanced models. Our Shopee data engineering team comes into the picture here — acquiring, cleaning and formatting data from various sources, which are used for our teams for analysis.

Beini (B): I enjoyed working with big data in university, so I wanted to secure an engineering role that allows me to use my skills to help business users. In my time working as a Shopee data engineer — I learnt that big data plays an important role in a tech company: improving efficiency and empowering our data scientists, analysts, and key decision-makers to make informed recommendations. So here I am, doing my part in the data engineering team!

So, what’s a typical day in the life of a Shopee Data Engineer like?

R:  We Shopee data engineers build capabilities to anticipate high amounts of big data for our big campaigns like 9.9, 11.11, and 12.12. Before each campaign, we’ll first estimate the resource needed based on expected traffic and orders, and then conduct tests using artificial data to identify potential bottlenecks. Following the test results, should bottlenecks occur, we make adjustments and recommendations for the volume of resources to allocate for campaign day. With all these preparations in place, we can then ensure that our data jobs can run smoothly during campaign season and that our stakeholders are able to find the data they require without any delay.

B: We also collaborate with various teams across the organisation to address their data needs in daily tasks. For example, with our Business Intelligence team to help them to optimise their resources with dashboards to monitor performance. Throughout the work day, you’ll often find me in meetings with our product managers and engineers to discuss business needs and user pain points, as we work towards creating new solutions and improvements together.

Interesting, so how has your journey as a Data Engineer been?

R: It’s been a great one! I actually started out at Shopee as a web developer. I worked closely with the Shopee Data Engineering team in those days  — they would have to ingest key business data, and as a web developer, I would help with the visualisation of key business data, with the use of various dashboards comprising charts and graphs, for easy reference. 

I always felt like Big Data was my calling even back then, though I was hesitant to pivot from web development to data engineering. I eventually decided to take a leap of faith and expressed my interest to my team lead — who as it turns out, was really encouraging about my decision to grow in this area. To facilitate a smooth integration process, I was paired with a senior Data Engineer who introduced the technologies used by the team and showed me the ropes. I’m truly grateful for Shopee’s support and guidance as I’ve been able to grow my career here across different domains.

B: I actually joined Shopee in its early days. Back then, there was zero to minimal data for us to tap into for analysis. Now, it’s great to see that we continued to build and refine our data insights throughout the years to create a Shopee experience that our users can enjoy across our markets in Asia, Europe and Latin America. From this, I’ve come to understand how big data works as a catalyst for business and technology.

What are some key projects that you’re proud to have worked on?

R: I’ve had opportunities to work on numerous exciting projects in my time here at Shopee. A particularly memorable one is when we had to build a platform for our business teams and senior management to observe our metrics such as order count, gross merchandise volume, and unique visitors, as the stats appear live. This way, our teams are able to understand status, view in-depth data analysis, forecast trends, and make better business decisions in real time. 

This is especially crucial for campaign seasons when we anticipate high traffic on the app. Finally, when it came down to our big campaign day, everything went smoothly on the platform. It was really a satisfying experience for us to witness the tangible impact of our work!

B: My team received feedback that our internal business teams were facing issues in generating data efficiently for them to review order performances on time. This hindered their ability to influence and create improvement and action plans. After working with them closely, my team was able to cut the time needed to generate the data outputs by several hours, which saved us significant calculation costs! It was wonderful to witness how our work is able to directly benefit our teams as well as the company.

What is it like working with your Data Engineering team here at Shopee?

R: Although our team of Shopee data engineers come from different backgrounds, we always focus on creating a team that cohesively works together and leaves space for feedback and discussion. I believe that building a tight-knit team is important to grow and achieve common goals together, so we often host casual team gatherings — such as our recent barbeque party and team visit to the Sea Aquarium. These have been great opportunities to form closer connections and foster team spirit.

B: My team participated in an escape room session for our team bonding, which was particularly memorable. We were able to enjoy our time brainstorming together to solve the mysteries, which helped us to break the ice with our newer colleagues. Getting to know each other better definitely helps us to communicate more effectively and freely! We also organised several bonding sessions across varying teams, where we were able to meet other engineers and product managers from other projects and domains — which helped us to collaborate even better when we partnered up for cross-team projects.

Want to know how Data Product Managers break down data for business teams’ use? Learn more from this video!

Photos were taken in compliance with local COVID-19 safety measures or before the COVID-19 pandemic.