The idea

This is an experiment. I’m testing ways to make the on-ramp for data analysis using R as easy as possible. Using R through posit.cloud in the browser immediately removes several complexity steps - open an account, open R, and you’re ready to go.

I’ve designed the slides so that, once you’ve run through the set up, you should be able to make a version of the same plot for whatever geographies you want (jump to this slide to skip all the R intro stuff and just make the plot). I imagine and hope it might be possible to go from zero R knowledge to a plot for your own region by the time the slides are done. And that could be something to build on - more self-contained, reproducible chunks of analysis and visualisation, reducing wheel reinvention.

That at least is the nice theory. I’ve run a couple of workshops using this approach, but those were only tasters. I could do with more feedback on what works and what doesn’t.

If you do have a go at this, I’d really appreciate any thoughts, successes, failures, anything learned from looking at this data for your own region.

The GVA factory method…

Let’s just mention Giles Wilkes’ point about the ‘GVA factory model’. Dividing GVA by jobs (or indeed by hours, as official productivity measures do; R example in the slides here!) I don’t think is the ‘horribly obvious maths’ he’s complaining about - instead, that’s what can happen if we treat these numbers in the way he parodies:

If you have three sectors, A, B and C, each with 10 workers, and their productivities are £100/worker, £50/worker and £20/worker , your strategy is to shift workers from C to B and A. For every worker you shift from C to A, you make £80 more a year – for free! Stop cutting hair, start making microchips!

The energy or telecoms sectors are prime examples: huge apparent output per worker, but their huge capital intensity isn’t directly visible in simple GVA x jobs data. GVA also spreads across value chains, accumulating in e.g. Apple’s Cupertino campus, not in the places iphones are assembled.

Fundamentally, output per worker going up is what we want - it directly measures how wealthy everyone is on average. ( ‘This is what we want’ is very open to question, though I think it still stands even if e.g. we impose ‘constrain so planetary boundaries are respected’).

But the streetlight effect is always present. Looking where ‘output per worker’ shines a light may lead us to badly misinterpret the pachyderm - that’s what I take Wilkes’ point to be. Do we have any option except to remain cautious and vigilant though? There isn’t any dataset or approach that can replace our need to think about how we use it.

That said, actually laying out ways to think about it in more detail would probably be useful…

Anyway, enough here to keep us busy. Here’s some data notes.

Data notes

• The ‘bres.gva.2d’ dataframe that the script loads in contains linked ONS regional GVA x sector data and BRES full time job count data.

• That’s processed here - if and when I make the data wrangling more legible, I’ll edit this and pretend it was well-organised all along. The prior steps include downloading/wrangling the latest (2023) GVA data here and bulk downloading the latest (2024) BRES data here.

• The pre-prepared data has a 3-year moving average applied - BRES data is volatile. So the plot is the 2021-23 average. It’s a better recent-history picture using that (edit the year in the script to look back as far as 2015-17).

• There are some fun¹ steps to harmonise the two:

  • The GVA data uses the 2025 ITL3 definitions. BRES doesn’t have those yet. But you can use local authorities from BRES - they mostly nest into 2025 ITL3, so can be summed where they don’t quite match. I made a function for checking geographical matches like this and getting automatic groupings for adding up jobs. It also tells you where things don’t nest - Arran in Scotland is the only tricky overlap that differs.

  • (There’s also one spelling mistake in BRES geographies that stops a correct match - “Rhondda Cynon Taff” should be “Rhondda Cynon Taf”. Make of that what you will.)

  • The regional GVA data has bespoke sector SIC code groupings - a different number for every different geographical level, with ITL3 having the fewest (disclosure reasons). So the BRES data has been binned into that same shorter sector list - 45 in total, though I’ve removed ‘households’ and ‘membership organisations’ as neither are present in BRES. I’ve then made some shorter names for nice plots.

That’s it! Any questions/issues, let me know at d dot olner at gmail dot com or message me on LinkedIn.

If using RStudio online: set up a posit.cloud account and create your RStudio project

Here’s the steps to get up and running through a browser using posit.cloud.

• Create an account at posit.cloud - follow that link and use the ‘don’t have an account? Sign up’ box. It’ll ask you to check your email for a verification link, so do that before proceeding.

• Once verified, go back to the posit.cloud tab and log in. You should see a message: “You are logged into Posit. Please select your destination”. Choose ‘Posit Cloud’ (not Posit Connect Cloud). That’ll take you to your online workspace.

• Click the new project button, top right

• Depending on what you want to do, either:

  • If you want the Tidyverse ready to go (but it’s a slightly older R version): select ‘new project from template’ (as in the pic below) and then “Data Analysis in R with the tidyverse” (if not already selected). This template comes pre-installed with the tidyverse package, which we’ll be using. Select then click OK down at the bottom. This will open your RStudio project where we’ll do the coding. NOTE: if I’m posting examples using posit cloud, I’ll let you know if the older R version is a problem. Mostly it’s fine.

  • Otherwise, just select ‘new RStudio project’ for an entirely blank instance of R where you’ll need to install all libraries (but it’s the latest R version).

Make a new R script and add a library

Now you should either be in RStudio online through posit.cloud or if using RStudio installed on your own computer, open that now. From here…

• Create a new R script by going to ‘file > new file > new R script’ (or using the CTRL+SHIFT+N shortcut). A new script will appear, currently just called ‘Untitled1’ until it’s saved for the first time.
• At this point, it should look something like this:

Let’s stop for a moment and look at the separate windows in RStudio.

• Bottom left is the console. Commands run here. You can test it by clicking in the console and trying a random command or two like those below (press enter to run a command in the console).

(Note that all code blocks in this post have a little ‘copy to clipboard’ icon in the top right when you hover, if you want to just copy the code for pasting into RStudio).

2+2

[1] 4

Or e.g.

sqrt(49)

[1] 7

• Bottom right of the RStudio window has various tabs, including local files (all kept inside your project folder so everything is self contained) and a list of available packages².

install.packages('tidyverse')

Now, whether in posit.cloud or on your own computer, you should have the tidyverse package available.

It now needs to be loaded as a library before we can use it:

• Put the following text at the top of the newly opened R script in the top left panel.

library(tidyverse)

When you’ve put that in, the script title will go red, showing it can now be saved (it should look something like the image below).

• Save the script either with the CTRL + S shortcut, or file > save. Give it whatever name you like, but note that it saves into your self-contained project folder.

Running code in an R script / loading the tidyverse library

All code will run in the console - what we do with scripts is just send our written code to the console. We do this in a couple of ways:

1. In your R script, if no code text is highlighted/selected, RStudio will Run the code line by line (or chunk by chunk - we’ll cover that in the taster session).
2. If a block of text is highlighted, the whole block will run. So e.g. if you select-all in a script and then run it, the entire script will run.

Let’s do #1: Run the code line by line.

• To test this, we’ll load the tidyverse library with the code we just pasted in (which is just one line of code at the moment!) Put the cursor at the end of the libary(tidyverse) line in the script (if it’s not there already), either with the mouse or keyboard. (Keyboard navigation is mostly the same as any other text editor like Word, but here’s a full shortcut list if useful.)
• Once there, either use the ‘run’ button, top right of the script window, or (much easier if you’re doing this repeatedly) press CTRL+ENTER to run it.

You should see the code get sent to the console, and a message like the one below confirming that R is ‘Attaching core tidyverse packages’. The tidyverse library is now loaded.

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.5
✔ forcats   1.0.1     ✔ stringr   1.5.2
✔ ggplot2   4.0.0     ✔ tibble    3.3.0
✔ lubridate 1.9.4     ✔ tidyr     1.3.1
✔ purrr     1.1.0     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

That’s it! Any questions/issues, let me know at d dot olner at gmail dot com or message me on LinkedIn.

Outputs:

RegEconTools website. All of this work comes under the broad heading of “Open regional economic data and tools”; this website is where I’m collating as much of this work as I can. Here’s the website repo.

Presentations / events

• South Yorkshire’s past, present and future: what does the data say?: presentation for the 1st SPERI seminar in a series about South Yorkshire’s political economy and history. Write-up on the Y-PERN blog here. 3.3.25.
• “Data action for local growth: what do we want to build?” Presentation to ONS subnational conference, Leeds. All the details and slides in this post. 14.11.24
• Y-PERN / SYMCA policy forum 1: GDP and beyond on the Y-PERN blog. 4.6.24
• Y-PERN / SYMCA policy forum 2: ‘High-Skilled Growth - The Importance of Universities in Driving Yorkshire’s Economy’ Word doc writeup. 24.10.24
• UPEN showcase: Regional Academic Policy Engagement in England (Y-PERN) - working with South Yorkshire Combined Mayoral Authority. Slide deck here. 2023

Techie talks

• Sheffield R Users’ Group presentation: “Making Economic Data Accessible”. Slide deck here.
• Sheffield Uni Data Stewards Event - presentation on using R to build pipelines for ONS and Companies House data, including links to various maps and plots.
• ONS Local Presents: “Using R for regional economic analysis - taster session”. The lovely people at ONS Local gave me a chance to do a whistlestop tour of using R for regional data analysis. Session outline here (including link to getting set up with R either online or on own machine). Course outline here. Online slides here. Both have links to ways to use R online and get the data. A recording of the session is here.

Presentations supporting the SY Plan for Good Growth:

• Slide deck for report to SYMCA on initial sector analysis findings highlighting structural change over time. (September 2023).
• Slide deck for presentation to SYMCA plus local authorities of Barnsely, Doncaster, Rotherham & Sheffield: sector growth, significance, productivity, gva vs jobs.
• Slide deck for SYMCA / Y-PERN joint meeting on growth and skills including SICSOC analysis showing which job skill level / sector combinations are significantly stronger or weaker between places.

Code for these is mostly in the ukcompare repo, including quarto/rmarkdown slides.

Bits and bobs

• Bootstrap estimates of the link between earnings and skills for South Yorkshire using the Annual Survey of Hours and Earnings and Census qualification data. Estimate plot here, code here. Used in the South Yorkshire Skills Strategy to give a central estimate: “If 10% of the population in South Yorkshire with Level 3 earned wages equivalent to those at Level 4 or above in South Yorkshire, total earnings could increase by an average of £200m.” (p.14)
• Job count sunburst interactive showing how SIC codes nest in South Yorkshire. Size of slice is number of jobs. Code here.

Data and code used

• For the 1971-81 ‘scarring’ work (used in ‘Steel City: Deindustrialisation and Peripheralisation in Sheffield’ with Jay Emery and Gwilym Pryce):
  • Harmonised Census data 1971 to 2011: country of birth and employment variables harmonised along with consistent geography. Full explanation in the readme there talks through how to get the data for country of birth (and further down the page there’s a link to the employment data). POSSIBLE TO-DO: HARMONISE WITH 2021 (and 1961???)
  • That data is used in this RMarkdown output that produces the plots used (from this repo with general data stitching code). The data for the RMarkdown output, using the harmonised datasets, is processed in this R Script.
• For the sector proportion plots, and other code on processing ONS regional GVA data for location quotients, mapping and other bits, see this code and data stepthrough on the regecontools site.
• The productivity “GVA vs JOBS percent change” plots and map don’t have a good walkthrough yet - the code (including code to update to latest BRES and regional GVA data) is here in the repo for the first tranche of sector analysis work done for SYMCA, and is fairly readable and self-contained there. That BRES data is automatically extracted using the super-useful NOMISR package in this script and processed in this script (where it’s linked to the LCREE dataset, along with GVA data - work done in this script and then collated for a report in this Quarto doc). TO-DO: MAKE WALKTHROUGH FOR PROD PLOTS
• The GVA per hour plot is part of this walkthrough on the regecontools page looking more broadly at some GVA per capita / per hour worked.
• The Beatty / Fothergill rank change plot is from this fuller breakdown of their data, with code walkthrough, on the regecontools site.

Other links

• The Y-PERN website.
• SYMCA Plan for Good Growth page, which includes the M-D economic analysis and my own sectoral 3-pager summary.
• Write up / blog post of the 2019 Sheffield Data for Good hack day.
• Liverpool City Region Civic Data Coop
• Story on Sheffield Neighbourhood Mapping project (link to current map version).
• Centre for Cities LA Evidential report.

References from the presentation:

• Rice / Venables: “The persistent consequences of adverse shocks: how the 1970s shaped UK regional inequality” here
• Sarah Willams, Data Action: Using Data for Public Good.
• Martin A. Schwartz, “The Importance of Stupidity in Scientific Research.” Journal of Cell Science 121.
• Peter Tennant on Bluesky talking about how we grow and why we need an open mind and be willing to be wrong.

Bits I didn’t manage to cram in the slides

• Analysis of ONS business demography data that links local authorities across the dataset, including business ‘efficiency’ (balance of births and deaths) showing something shifted in more recent years in the south. (I write about automating your way out of an Excel data hole for this project here
• The incredible Dutch secure data service data used in our Rotterdam project - paper here, supplementary material with a map here. Individual-level data! 100m^2, track over time! Link to other survey data! Secure, trustworthy, easy to use!
• Northern Irish Census data - summarised down to 100m^2. Allows you to e.g. see Belfast like this (interactive map).

Coupla lit bits

There are a couple of facts from my first head-butt of the literature that jump out. First-up are the basic demographic differences involved. Not only do migrants from outside the UK tend to be much younger, but there’s a difference in internal migration rates between ethnic groups too (though obv, best not to conflate ethnicity and external migration!) This is analysed in Finney/Simpson 2008. Their key finding is that, once demographics are controlled for, ethnic groups in the UK -

do not have a significantly different migration rate from the White Briton group when group composition is accounted for. [76]

They also mention, in passing, that:

accommodation that is privately rented is occupied by residents that are almost twice as likely to have migrated in the past year than the average resident.[72]

And vice-versa - home-owners are much less likely to move, a finding that’s consistent across all groups. The Fotheringham et al 2004 paper - a stupendous piece of work - looks just at out-migration rates (between 98 ‘family health service areas’) in England and Wales. They’d found -

a strong positive relationship between out-migration rates and the proportion of nonwhite population in an FHSA…

And, mechanism-wise, they saw two possibilities:

The generally positive relationship could be caused by the white population leaving areas of mixed race or by nonwhite populations having higher migration rates.

Finney/Simpson’s work suggests the latter - but that this is due to demographic differences. Also, Among the bzillion dynamics Fotheringham et al analyse, one that jumped out at me was:

Higher out-migration rates are associated with areas of high employment growth, suggesting a high turnover effect operates in such areas. That is, in-migration volumes will be high into such areas because of high employment growth, but recent migrants tend to be highly mobile and out-migration rates are therefore also likely to be high. [1666]

So we’ve got this high churn going on in economically attractive places - which also connects to the housing market, of course. More property-owning pushes an area away from this high-churn. And that could go both ways, couldn’t it? High turnover, from a Putnam perspective, could undermine some of social-capital formation and knock on to house prices. Those areas, if generally younger, might also be more urban, less desirable by older groups looking for family homes.

Or prices could be pushed up if people are piling in - but you can see equilibrium pressures at work as out-migration rates increase too.

Model pre-amble

Which segues me nicely into to the following silly little model. I’ve got a very long way to fully mapping out the dynamics involved but, here, I just wanted to get started with something very basic. This post is also an attempt to persuade R to do a simple little agent/stochastic model. I’ll wibble a bit at the end about the coding experience…

So this is a sorta-ABM with zero-intelligence agents making the simplest probabilistic moves. It looks like this:

• Nine hundred agents initially split evenly between three zones.
• All agents have the same 1 in 100 chance of deciding to move on every timestep…
  • Though that 1 in 100 chance is weighted slightly by the population of their current zone. If it’s more than an even proportion of the total population, their chance of wanting to move is increased slightly, and vice versa.
• Once an agent decides to move, they have a different function to choose where to move.
  • For two-thirds of them, they have no preference - they’ll decide to either stay, or move to one of the other two, with equal probability (but weighted by population).
  • One third of the agents, however, will have a preference for two of the zones (or a preference against one of them - same thing). This could be slight or large, depending on their preference set.

A few things to note before getting to the code:

1. The 1/3 agents are arbitrary - it’s 1 in 3 in each zone initially but it doesn’t matter. This is one aspect of the way the problem is thought about that I’d like to be sure I’m thinking straight on - if one (a) marks out a group of people as a specific sub-group and then (b) examine how that sub-group’s flows affect others’, might the result be an artifact of the labelling itself?

2. I haven’t dug into the original pieces of research in enough depth to make any sweeping statements. So, just to be clear, this piece isn’t in any way meant as a criticism of anyone else’s approach - it’s entirely just me thinking through some of the most trivially basic mechanisms that might be involved.

3. The population-weighted probability of moving I’m using is a way to push zone numbers back to equilibrium. It could stand in for any kind of pressure to move that agents might come under, from house prices to environment. I’m also aware there are plenty of mechanisms, in reality, that can make larger zones more attractive, not less, e.g. through Krugmanesque increasing returns feedbacks. The assumption here is that all those forces balance to a net-negative response to higher population.

4. Since I haven’t posted one of these little models for a while, I should point out that, not only do I think this kind of simple model is useful, I believe they can be extremely powerful and criticisms about lack of realism miss the point. See PhD chapter 3. But then I would say that, I suppose.

Right, that’s a lot of wiffle. On to…

The actual code. 1: Set up.

library(plyr)
library(reshape2)
library(ggplot2)
library(zoo)#For running means

‘Store’ just stores each timestep’s data for outputting once the model’s run:

#Store of time series data (to match how table gets converted to dataframe for rbinding)
#Time is iteration
store <- data.frame(zone = as.integer(), 
                    agent_type = as.integer(),
                    number_of_agents = as.integer(),
                    time = as.integer())

Then set per-zone population, each agent’s base probability of moving and the number of timesteps (though note below, I’ve hard-coded stuff that only works with 300 agents per zone… oops).

#population per zone
#(so they can be evenly distributed to zones to start with)
n = 300
#probability of an agent wanting to move on each turn
#1 in 100
p = 0.01
#iterations
ites = 1500

As mentioned, there are two agent types: two-thirds of agents don’t care where they move, if they’ve decided to move. The other third have a preference. These two preferences are set by giving each agent type its own vector for selecting a zone to move to.

This was one easy way of defining how a sub-group can be biased towards two zones: if, as here, their choice vector is 10 / 10 / 1, they only have a 1 in 21 chance of deciding to move to zone 3. (Note the range of other preferences for ‘bias’ in the comments.)

#even probability of choosing any zone (including my own)
even <- c(rep(1,each = 10),rep(2,each = 10),rep(3,each = 10))
#preference for zones one and two
#Slight preference
# bias <- c(rep(1,each = 10),rep(2,each = 10),rep(3,each = 8))
#Weaker preference
# bias <- c(rep(1,each = 10),rep(2,each = 10),rep(3,each = 3))
#Even weaker preference for one zone (thus stronger for other two)
bias <- c(rep(1,each = 10),rep(2,each = 10),rep(3,each = 1))
#Won't ever choose 3 (useful for testing assignment works)
# bias <- c(rep(1,each = 10),rep(2,each = 10)) 

Each ‘agent’ is just a row in a dataframe. Each row has an agent’s current zone, whether it’s going to move this turn and a reference to its preference (!). So it’s here that we set 2/3s of agents to ‘don’t care which zone’ (even) and 1/3 to ‘bias’.

A probability column gets added further below that determines their first decision - ‘shall I move this turn?’ Like most agent models, this is a little bit markov-chainy. I think.

#THE AGENTS:
#Keep them all in a single long dataframe
#assign agents to zones initially evenly
#One row per agent
#'moving' is flag: am I moving this turn?
#'prob': flag for which zone probability to use. 
#0 is even; 1 is biased
#Set a third of agents to prefer zones 1 and 2.
#Distribute them evenly between zones to start with
agents <- data.frame(zone = rep(1:3,each = n), 
                     moving = 0, 
                     prob = rep(c(0,0,1),times = n))#codes 2/3 majority

Just to show exactly what that creates: Zones 1 to 3 each have 300 agents in, and there are 200 who don’t care where they move to (0) and a hundred (1) that will use the ‘bias’ probability.

table(agents$zone, agents$prob)

   
      0   1
  1 200 100
  2 200 100
  3 200 100

Each timestep produces a ‘result’ table that summarises the number and type of agent per zone. Each of these ‘results’ goes into the ‘store’ dataframe for graphing later. But we need an initial ‘result’ to start with, as it’s used to work out how to weight moving probability on the next timestep - but the first timestep needs one too! So this one is just hard-coded to match the agent table above. I should probably work out how not to hard-code this. I’m not going to right now. So!

#WARNING: hard-coding the numbers for this first set of values based on 900 agents in total
#and a 2/3 majority
#So this matches store structure and initial agent state:
result <- data.frame(zone = rep(1:3, times = 2), 
                    agent_type = rep(0:1,each = 3),
                    number_of_agents = c(rep(200,each=3),rep(100,each=3)),
                    time = 0)

And that’s everything set up. On to ->

2: Running the model…

Here’s the model for-loop itself:

###########
# RUNRUNRUN
for (i in 1:ites) {

  #1. Weight probability of moving by population in each zone
  #Find zone population for this timestep
  zonepop <- aggregate(result$number_of_agents, by=list(result$zone),sum)
  
  #Sensible names for following the logic...
  colnames(zonepop) <- c('zone','population')
  
  #Weight probability of moving by population difference from even
  # zonepop$newprob <- (zonepop$x/(n)) * p
  #raise to power to make a larger effect (but 1 stays 1)
  zonepop$newprob <- ((zonepop$population/(n))^4) * p
  
  #drop any previous newprob column from agents
  agents$newprob <- NULL
  
  #Merge the probability for each zone into the agents
  #zonepop columns one and three is just 'zone' (for matching)
  #and the new probability of moving
  agents <- merge(agents, zonepop[,c(1,3)], by = 'zone')

This first section weights each agent’s probability of moving to the size of the zone they’re in. We know the populations are all even on the first step, so the initial ‘result’ above just uses the base probability, but on future steps it’s higher if more crowded, lower if less.

Note that the probability-calculating line raises the ‘zone population’/‘agent number’ ratio to the power of 4. This makes any deviation from an even population have an increasingly strong effect on agent’s likelihood of deciding to move (or stay, if the population’s lower than even.)

And then this just returns 1 if I’m deciding to move on this timestep:

  #2. Using weighted prob... Am I moving this turn?
  agents$moving <- rbinom(nrow(agents), 1 , agents$newprob)

You can see this produces roughly a 1 in 100 chance of each agent moving…

table(rbinom(nrow(agents), 1 , agents$newprob))

  0   1 
892   8 

But if population is higher in all zones (which it can’t be in the model, but just to illustrate). 10 times the current probability of 0.01 is about 1 in 10 deciding to move:

table(rbinom(nrow(agents), 1 , rep(10 * p,nrow(agents))))

  0   1 
823  77 

table(rbinom(nrow(agents), 1 , rep(10 * p,nrow(agents))))

  0   1 
803  97 

table(rbinom(nrow(agents), 1 , rep(10 * p,nrow(agents))))

  0   1 
813  87 

And if lower in all zones, agents are more likely to stay put:

table(rbinom(nrow(agents), 1 , rep(0.1 * p,nrow(agents))))

  0   1 
898   2 

table(rbinom(nrow(agents), 1 , rep(0.1 * p,nrow(agents))))

  0   1 
899   1 

table(rbinom(nrow(agents), 1 , rep(0.1 * p,nrow(agents))))

  0 
900 

For those who are moving, in this next step, they decide where to go. The fiddly part here are the two selections from the ‘even’ and ‘bias’ vectors that tell agents which zone they’re moving to. To explain it as much for my own later sanity as anything, here’s what’s going on. We’re just selecting a random index from each of them. In the case of ‘even’, 1,2 and 3 have the same chance of being chosen (as can be seen if we whack the number of random selections right up). Whereas ‘bias’ ends up telling about a tenth the number of biased agents to move to #3:

table(even[floor(runif(n = 1000000, min=1, max=length(even)+1))])

     1      2      3 
333529 332695 333776 

table(bias[floor(runif(n = 1000000, min=1, max=length(bias)+1))])

     1      2      3 
475498 477214  47288 

In the zone selection itself, each random selection is the same length as agents of that type. As the comments note, I’m a little amazed this works - I’m not really clear on how that random vector can be created and then, via an ifelse, be assigned to the correct index… oh well, it works!

  #3. If moving, where to?
  agents$zone <- ifelse(agents$moving == 1,#If I decided to move...
                        ifelse(agents$prob==0,#Move based on my preference of zone (or no preference)
                               even[floor(runif(n = nrow(agents[agents$prob==0,]), min=1, max=length(even)+1))],
                               bias[floor(runif(n = nrow(agents[agents$prob==1,]), min=1, max=length(bias)+1))]),
                        agents$zone
  )
  
  #Explanation for the zone selection above, since I'll probably forget.
  #Choose a random position from my (either even or biased/weighted) array of zone choices
  #Passing in a uniform random pick from each of the choice arrays
  #Of the correct length (the nrow subset)
  #It's still some form of witchcraft though - how does R know to distribute
  #the result to the correct index via the ifelse???

The result for this timestep is then stuck into the store for output later (also adding in a column to mark the current iteration). Converting a table to a data.frame reshapes it so it’s the right orientation to bind to the ongoing ‘store’ of results:

  #GET THE RESULT OF THIS ITERATION AND STORE IT
  #automatically reshapes, it turns out
  #so zone is in first column, zone pref type in second
  result <- as.data.frame(table(agents$zone,agents$prob))
  
  #Rename those fields to something sensible
  colnames(result) <- c('zone','agent_type','number_of_agents')
  
  #mark what iteration it is
  result$time <- i
  
  #Then add this step to the end of the data store
  store <- rbind(store, result)
  
}#end for

Display results

So that’s the results found. Now to show ’em. First-up, let’s add some extra data for total population per zone on each timestep:

#Find "total population in each zone at each iteration"
#Will be added as an extra bunch of rows to the output dataframe
#To fit the 'long' format ggplot wants
totpop_perzone_timestep <- aggregate(store$number_of_agents, by=list(store$zone, store$time), sum)

colnames(totpop_perzone_timestep) <- c("zone","time","number_of_agents")

#Make a new column for faceting the data.
#This one will be total population per zone
totpop_perzone_timestep$facet <- 'total pop'

#Relabel store's two agent types so each can have its own facet
store$facet <- 'agent-type: no pref'
store$facet[store$agent_type==1] <- 'agent-type: bias'

#Drop old agent_type column
store$agent_type <- NULL

#Stick 'em together in a new store
store2 <- rbind(store,totpop_perzone_timestep)

Now the data’s ready - just one nice little addition by combining ddply and rollmean from the zoo package to give us a running mean. This can help show the trend over time in a simple way. This sort of thing is really satisfying in R, when it works. One line! So ddply is applying the running mean for the number of agents in each zone/facet sub-group:

#Running mean...
smood <- ddply(store2, .(zone,facet), mutate, 
               rollingmean = rollmean(number_of_agents,250,fill = list(NA, NULL, NA)))


output <- ggplot(smood) +
  geom_line(data = smood, aes(x = time, y = number_of_agents, colour = zone), alpha = 0.3, size = 1) +
  geom_line(data = smood, aes(x = time, y = rollingmean, colour = zone)) +
  facet_wrap(~facet)

Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
ℹ Please use `linewidth` instead.

output

Warning: Removed 747 rows containing missing values or values outside the scale range
(`geom_line()`).

And there’s the basic result, then:

• The ‘bias’ agents (left-hand plot) move more to zones 1 and 2, as you’d expect.
• The ‘even’ agents in the middle plot, who (all other things being equal) don’t care where they go, end up having a larger number in zone 3 as they respond to the pressure of increasing population. Remember, that pressure is only coming from one-third of the agents.
• Overall, zones 1 and 2 end up with higher total populations because of the minority groups’ preferences.

I’ve not tested whether/at what point the ‘bias’ agents’ preference function wouldn’t outweigh the population push but, here at least, their large preference for those two zones wins out.

To look at it from another perspective, the following re-jigs the data so we have the proportion of ‘even’ vs ‘bias’ agents in each zone:

#Different output for looking at the proportion change of the two groups in each zone
#Use the original 'store' for this
#Convert wide so that each agent type has its own column
#To make finding proportion  per zone easier
proportions <- dcast(store, zone+time ~ facet, value.var = "number_of_agents")
 
#Proportion of majority as percentage of whole
proportions$percent_majority <- (proportions$`agent-type: no pref`/
                       (proportions$`agent-type: bias`+proportions$`agent-type: no pref`))*100

output <- ggplot(proportions, aes(x = time, y = percent_majority, colour = zone)) +
  geom_line() 

output

Zones 1 and 2 see a lot of ‘even-agent flight’, it seems. Which makes perfect sense given the trivially simple dynamic: it’s nothing more than a response to some equilibrium pressure as one group who prefers a zone (for whatever reason) decides to move there.

All agents, regardless of type, are affected in the same way by population pressure: their decision to move is the same. They only differ in where they prefer to move to. Many of the ‘bias’ group prefer to move somewhere in the same zone or a similar one.

I’m really labouring the point now, I know, but… the point being, assigning causality here is a little murky. Without the ‘bias’ groups’ preference, the ‘evens’ wouldn’t end up dominating zone 3.

This is, in a way, just a slightly different take on the Schelling segregation dynamic, except that it’s not about people’s preferences for any particular type of neighbour, but rather some people’s preferences for particular places, and what the knock-on effects of that could be.

Random coding wibble

So that’s enough ill-informed migration wiffle. On to coding wibble. That was in some ways amazingly easy to set up, and R does some things just beautifully. The tricky part: I went away for a month and then it took me about two hours of staring to figure out how it worked. I fixed that by naming variables sensibly. Phew.

As far as I was able to figure, there was no way to circumvent the main for-loop and subsequently it’s pretty slow. “As far as I was able to figure” isn’t very much, so perhaps there’s a way of making this more R-native - though I suspect the kind of non-ergodic timestep processes that drive ABM might not be R’s forte. Though though: the slow part is actually the rbinding and table-making, so there may be another way.

On the plus side, I can write it up like this in RMarkdown… though perhaps Python will let me do that too, if I can get round to trying it. And my first thought on coming back to the model after a break: if this were Java, it would be perhaps make a lot more sense right now, and running faster. OOP and ABM go together: it’s very easy to see what the agents are. Here, pleasing in its brevity but challenging to keep all the working parts in mind.