Melbourne is the beating heart of Australia's quietly surging fintech industry and for almost a decade, local tech-enabled trading company Pepperstone has been carving its own niche, developing a burgeoning startup into one of the largest Forex and CFD brokers in the world.
The company's rapid growth presented challenges when managing scaling issues and as a consequence, Pepperstone turned to Apache Kafka® to assist in migrating data-intensive, batch-oriented processes over to event-based architecture.
Pepperstone Team Lead Douglas Reith took the time to chat about the early teething pains that came with rapid business growth and how Factor House's Kpow toolkit continues to support their innovative expansion into the international financial services marketplace.
Who is Pepperstone?
Founded in 2010, Pepperstone was introduced to the world amidst the steep trajectory of rapid globalisation, particularly across the tech-trading space, following the GFC only a couple of years prior. Many enterprises were beginning to find their feet again and the opportunity presented itself to create a world where ambitious traders could embrace the challenges and opportunities presented by global market trends.
At its heart, Pepperstone is an online Forex and CFD broker, but much more than that, the platform adopts a digitally-led solution that empowers clients to trade in new and innovative ways.
Now, Pepperstone is one of the world's largest MetaTrader brokers, processing an average of US$12.55 billion in trades every day on more than US$9.2 billion retail trading accounts. The company has received multiple awards for innovation and technology, offering clients highly competitive trading prices, unprecedented speed, and renowned customer service.
Addressing the Challenges of Rapid Evolution
For a rapidly evolving tech platform, managing workflow across the business with an increasingly integrated reliance on Apache Kafka without external support was fast becoming a difficult and often laborious task. The levels of automation that would have been necessary to keep up with the demand on internal resources simply did not exist.
Integral to the delivery of the suite of Pepperstone's client services, Kafka has become central to the evolution of their technology. However, in the early years, limited availability of specialist support led to a bottleneck of workflow and an often cumbersome inability to turnaround quick fixes and improvements. Senior engineers were unable to progress much of their own priorities, whilst swamped with tasks that few in the broader team were able to 'self serve' using the existing core software.
Apache Kafka is globally recognised as a reliable and sophisticated distributed streaming platform. However, it does not offer much tooling when it comes to providing whole system visibility. This meant that the Pepperstone team often found it difficult to review the current state with any real confidence as to what was happening at any given time, in order to plan for future growth or proposed projects.
Meet Factor House and Kpow
Factor House caught the Pepperstone team's eye at the VoxxedDays conference in 2019, where Factor House was presenting the Apache Kafka Three Ways (AK3X) workshop and signing up Beta testers for their Kafka toolkit Kpow. Following an introduction, the Factor House team delivered a tailored AK3X workshop to the Pepperstone engineers, who trialled the platform soon after within their staging environment. As they had not been long into their Kafka journey, there was a long list of quick wins achieved through the collaboration.
"Initially we would open up Kpow to check topics and consumer groups after the deployment of a new application or if we wanted to get some statistics. Now, we are doing that as well as sampling data, assessing capacity requirements and troubleshooting issues."
The Factor House team worked closely to onboard the team and have continued with the same 'always on' approach.
A 360 Degree Visual Awareness
The introduction of the Kpow Kafka console immediately empowered Pepperstone to access all of its key Kafka metrics by providing 360-degree visual awareness of their system. This functionality was critical to delivering an exponential improvement in the degree of time and resources required to both resolve tasks and support innovation.
According to Pepperstone Team Lead Douglas Reith:
"Kpow has improved how we communicate Kafka to the rest of the IT team and the rest of the business. It has helped us assess what we may need to do in the future as well as identify issues related to development and infrastructure standards".
Over time, Kpow has become even more 'featureful'. Reith explains that by "deploying it in our ecosystem, the platform has given us a much greater perspective about Kafka and how it and our whole system is operating at any given time. It means that if I want to see anything in our cluster, I can now go to Kpow first, rather than establishing an internal command-line job".
Pepperstone is a Kpow enterprise client, and the collaboration continues to grow from strength to strength. Reith added, "We would recommend it to other companies to raise the awareness and visibility of Apache Kafka as well as assist data engineers in their work".
Manage, Monitor and Learn Kafka with Kpow by Factor House.
We know how easy Apache Kafka® can be with the right tools. We built Kpow to make the developer experience with Kafka simple and enjoyable, and to save businesses time and money while growing their Kafka expertise.
Kpow is compatible with Apache Kafka+1.0, Amazon MSK, Instaclustr, Aiven, Red Hat AMQ Streams, Vectorized, Azure Event Hubs, Confluent Platform, and Confluent Cloud.
Start with a free 30-day trial and solve your Kafka issues within minutes.
Kylie Troy-West is a Co-Founder and COO of Factor House. Her focus is empowering the Factor House team to deliver their vision of world-class tools for Apache Kafka.