Research Publishing Analytics

Assessing my track record using analytics

1/ At my employer the University of Melbourne, I work with Early and Mid Career researchers on their competitive grant, research program and publishing activities. Several weeks ago a colleague commented on a draft website list that included Academia.edu and ResearchGate: two platforms where academics can post the pre-publication accepted versions of their journal articles for public readers. During the exchange I revealed that I had a paid Academia.edu subscription. The exchange got me thinking further about research publishing analytics - and my own track record (relative to opportunity) in particular.

2/ In 2014 whilst in Victoria University’s Office for Research, I attempted to first formulate an Academic Moneyball approach to research performance analytics. The approach acknowledged the Michael Lewis nonfiction book about how the former Oakland A’s coach Billy Beane used sabermetrics - a form of statistical analysis - to turnaround and build a high performance team. My approach modelled research performance as both a set of activities and the present (discounted) value of future cashflows. I began to see competitive grant applications as like options in financial markets or real options in finance and innovation. I trained a few colleagues in my approach using Google Scholar publication lists to assess the track records for ARC DECRA grant applicants.

3/ Google Scholar and Academia.edu analytics provided a further way to model and understand a publishing track record. Google Scholar was able to find and collate citations over time from international sources. Academia.edu added the further functionality of identifying readers and their institutional affiliations (if they were Academia.edu subscribers), and collating download statistics. From these two platforms it was possible to identify which articles had significant end-user impacts in terms of citations, downloads, and readers. There were limits (Academia.edu’s spam and its heavy use by graduate students instead of faculty) but this was also a very affordable way to gather research publishing analytics.

4/ These platforms revealed some surprises in my own research track record. I have a long tail of Disinformation dossiers, magazine articles, and Masters essays that have been cited. Some of what I feel is my best academic work has few or no citations at all. The two standout pieces are a 2009 co-authored paper with Ben Eltham on Twitter’s role in Iran’s election crisis, and a 2007 draft of a Smart Internet Technology Cooperative Research Centre project on the late Clayton M. Christensen’s disruptive innovation framework. Both emerged from crises: the conference paper was dropped from the refereed conference stream because it mentioned information warfare and psychological operations; the SITCRC report exists in draft form and was never formally cleared for publication. Both are ‘hot hand’ examples of ‘winning streaks’: both absolute and relative out-performance compared to my other publications.

5/ Using analytics to look at a publishing track record this way enables more economy of effort and scalability. The ‘Twitter Free Iran’ article for example was written in the three weeks after the events it covered and before the subsequent big data analysis of subsequent articles. We used Graham Allison’s co-authored Essence of Decision study on the Cuban Missile Crisis as a writing model. During the SITCRC piece I discovered agile software development, took Scrum certification training, and spent a lot of time in a local Borders store coming to grips with design patterns, refactoring, pair programming and other innovations. These research experiences and models are in part why these two publications have been unexpected successes.

6/ An analytics-based approach changes how publications are conceptualised and developed in a research program. It introduces a feedback loop into the process: a shift closer to market campaign planning and monitoring. This is a demand-driven approach: rather than writing on rapidly changing ‘hot topics’ and subjective interests, you instead write on topics and approaches that seem to generate a more spontaneous readership. This is a demand-driven approach. It introduces more discipline into the writing process - similar to the world-building skills that novelists need for creating a branded multi-volume series. It also means making decisions about what not to write.

7/ Looking at my Google Scholar profile I see several patterns in terms of citations. (1) a sequence of papers from the early aughties as I was first beginning to write for academic audiences and transitioning from a career in freelance journalism; (2) several Masters essays from Swinburne University’s former strategic foresight program which have been cited (and plagiarised); (3) sequence of papers from 2006-09 when I established an independent and self-funded research career outside the Smart Internet Technology CRC; (4) a few papers on journalism that were basically invitations from Queensland University of Technology professor Axel Bruns to contribute to those debates; and (5) Ben Eltham and I have a few co-authored papers that continue to be cited. (6) There is also a 6 year gap from 2014 to 2020 when I worked in research administration and completed my PhD.

8/ You can see from my research publications trajectory that it co-evolved with several important institutional experiences. Swinburne University’s strategic foresight program and the Smart Internet Technology CRC were crucial to my earlier career, but both were relatively short-lived as institutions. My collaboration with Ben Eltham between 2009 and 2014 was more productive in terms of the number and the diverse range of papers produced. Axel Bruns was an important mentor. These collaborative relationships underpin periods of higher research productivity that the research publishing analytics capture.

9/ Cross-referencing my Google Scholar and Academia.edu profile enables me to narrow down this list even further. The standouts are the Twitter Free Iran paper co-authored with Ben Eltham and a further paper on Australian screen policy; the Smart Internet 2010 (2005) report with the Smart Internet Technology CRC; two papers that emerged from after the SITCRC’s Disruptive Internet project on disruptive innovation; my 2006 MA thesis on Herman Kahn and North Korea; and a 2014 co-authored paper with Eltham on Australia’s defence and national security policymaking that emerged in part from my PhD research on strategic culture. Condensing this further leads to three areas: (1) crisis and event studies; (2) disruptive innovation critiques; and (3) nuclear strategy and strategic culture. These seven or so publications make up 80% of my Academia.edu downloads.

10/ I’m now thinking about what next to write about in the post PhD phase of my research program. I have extensive working notes from my PhD and several planned articles to write on strategic culture and terrorism studies. I will be targeting higher ranked journals such as Contemporary Security Policy and the Journal of Strategic Studies. The crisis and event studies approach of the ‘Twitter Free Iran’ paper could be replicated in future studies and methodologically updated with more rigorous data collection. With the passing of Clayton M. Christensen, in the future I may also revisit and update the disruptive innovation material as a sociological study. Using analytics helps to identify and refine where I can concentrate my writing efforts in order to have the most significant research impacts.