Swift is an open-source, highly available, distributed, eventually-consistent object storage system. As the project has matured, so has the need to store more and more data. Objects in swift are tracked in the containers they are placed and a container in essence is a SQLite database which is distributed throughout the cluster. As the number of objects in a container increases, so does latency meaning larger sites have moved to using expensive solid state drives (SSDs) for container storage, but there has to be a better way we can solve this in software, right?
For this presentation we'll cover the design decisions that have enabled Swift's success, and then show how these same decisions need to be revisited as we cater for the ever increasing size of containers. We'll consider a proof of concept container sharding implementation that attempts to address a few of these limitations, allowing for larger datasets to be deployed on cheaper commodity hardware, splitting a container database over many databases and nodes.
We won't just present the final solution, but we'll iterate over the different approaches tried, so that you can learn from our mistakes and lessons learnt.
Matthew is a software engineer working at Rackspace Australia, more specifically Rackspace Private Cloud where he primarily works on upstream OpenStack as a Swift core. Based in Melbourne, he spends his time hacking on Swift. Matthew was the co-founder of the Kororaa Linux distribution which has given him careers in both Linux system administration and software development. During his time with Rackspace he has worked both as an Enterprise Linux Sysadmin based in the UK and is now back at his roots as software developer in Australia.
Geelong is Victoria's second largest city, located on Corio Bay, and within a short drive from popular beach-front communities on the Bellarine Peninsula as well as being the gateway to the famous Great Ocean Road
linux.conf.au is widely regarded by delegates as one of the best community run Linux conferences worldwide and is the largest Linux and Open Source Software conference in the Asia-Pacific.
Our Sponsors help make linux.conf.au become the awesome conference everyone comes back to year after year. Come see who's on board this year, or find out how to get in contact with us