A lot has changed since 1964 when Pickles started life as a Stock and Station Agent in Moree, in rural northern New South Wales.
Over the last 57 years, Pickles has expanded across Australia and Asia, evolving into Australia's leading auction marketplace specialising in asset remarketing across industrial, cars, trucks, and salvage - selling 228,000 vehicles and over $460 million in transport and construction equipment in the last financial year.
Customer and Technology Obsessed
Pickles are a value-driven company. At the heart of everything they do is customer experience, quality relationships, and innovation -- they seek to find simple solutions to complex problems and deliver a superior purchasing experience to their buyers.
In the decade since Apache Kafka® was open-sourced in 2011, it has become essential to businesses who strive to understand their customers and react to their needs in real-time. It will not surprise that Kafka is a core technology of Pickles' real-time online marketplaces.
Solutions Architect Krishnakumar Arjunan is one of 20 engineers in the Pickles development team that spans Australia and India. We had a quick chat with Krishnakumar to find out how Kpow helps the Pickles team deliver.
Innovation and Evolution
Innovation and evolution are central to the Pickles delivery model - which is constantly evolving. Pickles introduced remote bidding in 2002, and by 2006, PicklesLIVE, their real-time online auction service - which allows customers to participate at any of their live auctions remotely -- passed the 100,000 vehicles sold online milestone. This service has gone on to become one of their most popular offerings.
Complimenting this service is PicklesONLINE, an innovative bidding system for online auctions that run over several days. Customers indicate their maximum bid amount, and the PicklesONLINE system then bids on your behalf up to that amount, but not beyond it.
The Heart of the Problem
Easy access to data on topics is key to validating the expectations of systems built with Kafka. The Pickles engineering team found simple tasks like validating message delivery could sometimes take hours. It was such a common problem that they even began building custom tooling to read topics and check data.
"We used a custom microservice to read data from a topic and validate message delivery, but we can't build microservices to check each use case"
- Krishnakumar Arjunan
Data Inspect is Kpow's most popular feature because it takes the pain out of ad-hoc Kafka queries. Configurable serdes with full Avro support and JQ-like filters for searching topics allow engineers to navigate topics and locate messages, with the added benefits of role-based access control and data masking for sensitive data. All of which are vital for Pickles.
"Kpow is intuitive and user-friendly. It saves us time, is efficient, and our developers like using it," Krishna says. "We can view messages with Data Inspect as soon as a message is persisted. We can also check the schemas, see if their names and versions are appropriate and that the consumer group names are unique".
Engineer Focused Tooling
After months of searching for tools, one of Krishna's Australian team members heard about Factor House's Kafka toolkit, Kpow. Pickles opted for a free trial in January 2020, and by March, had signed on for the paid subscription. It gives them an eagle-eye view of their Kafka ecosystem and allows them to get right down into the details to pinpoint issues - of particular interest is the ever-present question of consumer lag which is presented directly in the Kpow UI.
The toolkit is also rapidly improving, with 59 updates within the first year of its release. Factor House is continuously incorporating user feedback into product development. Krishna says there's always been a fast turnaround time for any of his or his team members' queries and feature requests.
"It's awesome. We connect with the Factor House team, and someone helps me within 24 to 48 hours if it's a small one (request) and if it's bigger within a week. They're very friendly and make sure the problems are solved."
Evolve with Apache Kafka
Pickles is an excellent example of a company leveraging Apache Kafka to meet their customers' dynamic and ever-changing needs, and Kpow has become the go-to tool for Pickles to sort out their Kafka issues.
"If there's a problem with our Kafka system, there's no way to track it unless you have Kpow. If you know Kafka, you should be able to use Kpow without an issue"
- Krishnakumar Arjunan
The evolution of Pickles reaches much further than their shift towards live and real-time online auction marketplaces. In 2018, Pickles launched Pickles Ventures, an incubator where start-ups, entrepreneurs, and corporates can leverage Pickles' expertise and resources to bring new solutions to the market.
To find out more about the evolution of Pickles Auctions and Pickles Ventures head to pickles.com.au.
Manage, Monitor and Learn Apache Kafka with Kpow by Factor House.
We know how easy Apache Kafka® can be with the right tools. We built Kpow to make the developer experience with Kafka simple and enjoyable, and to save businesses time and money while growing their Kafka expertise. A single Docker container or JAR file that installs in minutes, Kpow's unique Kafka UI gives you instant visibility of your clusters and immediate access to your data.
Kpow is compatible with Apache Kafka+1.0, Amazon MSK, Red Hat AMQ Streams, Instaclustr, Aiven, Vectorized, Azure Event Hubs, Confluent Platform, and Confluent Cloud.
Start with a free 30-day trial and solve your Kafka issues within minutes.
Kylie Troy-West is a Co-Founder and COO of Factor House. Her focus is empowering the Factor House team to deliver their vision of world-class tools for Apache Kafka.