System Administrator at Scrapinghub (Cork, Ireland) (allows remote)
Job Description
Scrapinghub is looking for a senior systems engineer to join the team. This role will be responsible for the successful operations and scaling of the infrastructure and software that powers crawls of over 2 billion pages a month.
Founded by the creators of Scrapy, Scrapinghub helps companies turn web content into useful data with a cloud-based web crawling platform, off-the-shelf datasets, and turn-key web scraping services.
Join us in making the world a better place for web crawlers with top talented engineers working remotely from over 30 countries.
Your key responsibilities will be to:
- Write tools and scripts to provide automation and self service solutions for ourselves and other teams
- Design new systems to support production services
- Creatively solve scale challenges regarding a rapidly expanding cloud environment
- Help improve monitoring and identify key performance metrics
- Proactive RD - discovering and implementing new tools, emerging technology, etc.
- Disaster recovery design, implementation, and maintenance
- Troubleshooting and resolution of server/network issues
A few examples of things you’ll do:
- Migration of Cloudera Distribution for Hadoop (CDH) from version 4 to version 5 and the 50+ TB of data stored inside it, with minimal downtime
- Building and optimizing a Elasticsearch+Logstash+Kibana stack for our development team to monitor and analyze production system usage.
- Design and implement a continuous integration and deployment system based on Docker, Mesos and an automatically configured http load balancer able to reroute traffic in case application containers die.
- Automate servers setup to scale to +300 servers on cloud providers and bare metal, be ready to replace hardware at any time without service outage.
- Setup and optimize a high available multi master MysqlDB and RabbitMQ cluster
Skills Requirements
Our Ideal Candidate would have:
- 3+ years recent experience working as a Linux administrator/engineer at reasonable scale (dozens of systems) and designing/deploying ‘highly available’ solutions
- 2+ years of recent professional experience designing, developing, and operating Configuration Management solutions such as Chef, Puppet, Salt (preferred), or Ansible (preferred)
- Availability to work on European time zones
- Solid experience in Linux tuning, profiling, and monitoring
- Strong skills in at least one language: Python (preferred), Ruby, Bash
Bonus points for:
- Experience deploying/managing Docker and LXC
- Solid understanding of cloud/networking/distributed computing environment concepts; including TCP/ IP connections, firewalls, VLANs, etc.
- Experience and understanding of contemporary metrics, monitors, and logging solutions especially ELK, cadvisor, Grafana, influxdb, etc..
- Highly organized, able to multi-task, able to work individually, as well as within a team, and across teams
- Excellent written English skills
- Working knowledge of load balancing technologies
- Proven experience working remotely and collaborating in a cross functional team environment
- Familiarity with software engineering practices, including n-tier architecture, configuration management, development methodologies (e.g. agile, waterfall, spiral, prototyping), etc.
About Scrapinghub
Scrapinghub is a startup with the goal of providing the best web scraping technology.
We currently provide services for running Scrapy web crawlers, storing and searching crawled data, visualizing the crawl process, automatic information extraction (based on supervised learning) and a proxy network for routing requests. We also develop open source libraries for web crawling and information extraction.
Our clients are from a diverse range of industries, they're usually technical and build very interesting products with the data and services we provide.
This is an opportunity to join at an early stage where you can have a huge impact on the success of the company.
Joel Test score: 11 out of 12
The Joel Test is a twelve-question measure of the quality of a software team.
- Do you use source control?
- Can you make a build in one step?
- Do you make daily builds?
- Do you have a bug database?
- Do you fix bugs before writing new code?
- Do you have an up-to-date schedule?
- Do you have a spec?
- Do programmers have quiet working conditions?
- Do you use the best tools money can buy?
- Do you have testers?
- Do new candidates write code during their interview?
- Do you do hallway usability testing?
Visit the Scrapinghub company page
return to search results view all Scrapinghub job listings
Leave a Reply
You must be logged in to post a comment.