Fortunately, I had recently read a book overcome exactly this type of problem. It was called Big Data Analysis with R by Simon Walkowiak, which explains a number of different strategies for analysing ‘larger than memory’ datasets with R.
I learned from that book that the easiest way for me to do what I wanted to do was to use the RSQLite package to write a script that would create an SQLite database on my computer’s hard drive. Then europe rcs data I could write all of the EPC data to this database programmatically from within my R session. Next I was able to send SQL queries to my new database from R which summarised the data from each local authority one at a time. This process meant I could answer my research question without having to load the entire database into memory at once. I was also helped in doing this by another book which I’d read beforehand which was called Teach Yourself SQL in 10 minutes by Ben Forta, as that taught me how to write the SQL queries which I needed to use to communicate with my new database.
This piece of research went on to be included in a report which IF subsequently published called Rabbit Hutch Homes: The growth of micro-homes (January 2020), which achieved a significant amount of press coverage and was submitted to several government inquiries on improving Britain’s housing supply.