Practical approaches for web scraping for research

It is no exaggeration to say that the web is a fertile source of data - offering deep insights into people’s beliefs, opinions, transactions, movements and many other aspects of their lives.

For social science academics and data scientists, the UK’s legal environment appears (although not definitively) to provide opportunities to capture these data at scale in service of research goals.

Referencing UBDC’s project and open-source software platform to scrape short-term-let data from Airbnb, this webinar provided practical guidance on how researchers, technologists and data scientists can approach web scraping, from the selection of online sources to the planning, conceptualisation, governance, risk management and implementation of technical approaches.

Throughout the session, a series of practical examples covered data scraping using UBDC’s established scraping method from Airbnb’s online platform.

Resources

Practical approaches for web scraping for research – using Airbnb as an example data provider

‍Code examples, documentation and practical exercises used in the webinar on GitHub