Policy Journal Trends and
Tensions: JPAM and PSJ

William C. Adams
Donna Lind Infeld
Michael Ruddell
Laura Minicelli


ABSTRACT


Academic journals may especially influence the development of an emerging field; early editors of public policy journals were explicit about that goal. Two leading journals – the Journal of Policy Analysis and Management and the Policy Studies Journal – were compared for changes between the early 1980s and 2007-2010. Over time, both journals published far fewer research articles by practitioners, and the non-university share of both editorial boards also declined. The journals continued to focus largely on the United States. Both showed a dramatic increase in the proportion of co-authored articles. Over time, JPAM became far more likely to publish studies by economists and far less likely to publish political scientists while PSJ increasingly published political scientists. JPAM authors now primarily reference economics journals. PSJ authors often cite political science, public administration, and other public policy journals. Both journals moved away from broad policy essays, with JPAM heavily trending to multivariate (essentially econometric) secondary analyses of large data sets and PSJ including a broader range of methodologies. Contrary to early predictions of a progressively more interdisciplinary field, the opposite trend – stronger alignments with specific disciplines – reflects public policy's ongoing challenge in transcending long-standing academic legacies and boundaries.

KEYWORDS: Public policy journals, authorship, editorial boards, interdisciplinary research, publication trends.


Introduction


Publishing peer-reviewed research allows academic journals to help shape a field by setting its boundaries, sustaining or discarding certain lines of investigation while introducing new ones, featuring studies that reinforce or advance preferred methodologies, and screening out papers judged unworthy. In an emerging interdisciplinary area, leading journals may play an especially influential role.

Public policy is one such field, a notoriously tricky academic arena with intersecting disciplines, varied methods, and diverse foci. Perhaps public policy journals have a particularly influential role molding, and not just reflecting, such a field. Indeed the "main function" of the Journal of Policy Analysis and Management, according to its first editor Raymond Vernon (p 578, 1985), was to shape developments that would "speed the day" when public policy would be a "recognized professional field" with "a hard core of methodologies, a considerable body of accumulated wisdom, and a common language for the communication of such ideas." Other early editors also made it clear that their journals' proactive task was to "help give some shape to the developing area of policy studies" (Nagel, 1973 in Policy Studies Journal) and "nurture and structure a discipline" (Brewer, 1974, p. 239, in Policy Sciences).

As gatekeepers, editors and reviewers must continually decide what research to reward with publication. The cumulative impact of their individual verdicts builds an overall voice and tone for a journal, constructing a highly visible component of a new field such as public policy. Although constrained by the choices among submitted papers, much of a journal's power and prestige is due to the selectivity of its choices, winnowed to only about two dozen research articles each year. A general public policy journal cannot possibly cover all topics, all countries, and all methods, but the final publication patterns – whether inadvertent or due to intentional priorities – still allocate scarce resources and showcase the chosen few articles as the most exemplary research in public policy.

Whether or not the editor explicitly states that the journal's role is to shape the field, the quarterly compilation of all their incremental choices reflects priorities on many dimensions. To what extent, for example, does it publish research that is grounded in particular parent disciplines such as economics or political science, that relies on certain preferred methodologies, that takes a practitioner or more academic orientation, that focuses on the U.S. or elsewhere in the world, and that requires increasingly large collaborations or can be conducted by a solitary scholar?

Journal content matters if, over time, these judgments make journal decision-makers architects rather than merely curators of the field. Thus, in areas related to public policy, scholars have found it useful to track trends of research articles in their leading publications, including economics (Heck and Zaleski, 1991), political science (Bennett, Barth, & Rutherford, 2003; Elliott, Ho and Holmes, 2009), public administration (Lan and Anders, 2000; Raadschelders & Lee, 2011), and management (Scandura and Williams, 2000). While not a longitudinal study, deLeon et al. (2010) reviewed recent years in Policy Studies Journal. Otherwise, systematic reviews of trends in public policy journals could not be found.

To explore public policy journals in more detail, some screening is required. Which public policy journals matter most? Without sinking too deep in the quagmire of competing citation and reputation measures of journal rankings (Beed & Beed, 1996; Pontille & Torny, 2010), a few general conclusions can be drawn. Exact standings vary year-to-year and measure to measure, but the most highly ranked, primarily American, public policy journals without a subject-area focus typically are (in alphabetical order):

  • Journal of Policy Analysis and Management
  • Policy Sciences
  • Policy Studies Journal

Public Choice, with its distinctive theoretical base, is regularly in the top tier as well.

With more resources, one might track trends in all these journals, but this undertaking first narrowed the pool based on 2011 Eigenfactor Scores in the online ISI Journal Citation Reports. These scores are calculated from the number of citations over the past five years (excluding each journal's self-citations) weighted by the citation prominence of each citing journal. Using that final criterion, the Journal of Policy Analysis and Management (JPAM), and Policy Studies Journal (PSJ) were chosen for more extensive analysis,1 although future research could examine Policy Sciences, Public Choice, and other journals as well.

To assess the directions of these two leading journals, we coded 400 research articles on a variety of variables. Focusing on full-fledged research articles was the practice of prior journal studies for related disciplines and allows for comparability between JPAM and PSJ. Excluded were book reviews, book notes, editor's notes, announcements, addresses, and organizational news sometimes found in both journals. Also excluded from the comparison of research articles were the usually short pieces sometimes appearing in JPAM (after the section entitled "research articles") under such headings as "point/counterpoint," "insight," "reflexions," "professional practice," and curriculum, along with short JPAM "case notes" and its lists of working papers and dissertations. Altogether these later sections constituted no more than one fifth of JPAM's pages during the period studied, but to be comprehensive, we will also report their content in the analysis that follows. In reviewing PSJ's research articles, we excluded short comments and replies that occasionally followed articles.

To compare trends over time, a good period to start is the early 1980s. JPAM was launched in 1981 as the field of policy research was rapidly gaining academic standing. For each journal, the first 100 research articles were reviewed starting with their first issues in 1981.2 For the later period, research articles were reviewed for each journal ending with the last issue in 2010, working back to assemble 100. The later period encompassed all of 2008-2010 and part of 2007 for both journals. The initial period included most of 1981-1982 for both journals and extended into part of 1985 for JPAM since PSJ was then including more research articles per issue.

Closely reviewing these 400 articles was a time-consuming process encompassing more variables than reported in this summary. The coding was refined during repeated group discussions and the resulting inter-coder reliability was 96%. Data from these 400 articles offer an opportunity to gauge trends in these two important journals across nearly three decades.

Over time, how have these public policy journals focused our attention? We first examine three issues:

(1) Are they directed inward to the academic world or outward toward practitioners?
(2) Are they emphasizing the United States alone or other polities as well?
(3) Are more research articles authored by teams rather than by individuals?

Then we turn to two perhaps more controversial questions regarding public policy's disciplinary and methodological struggles:

(4) Have these leading journals become more interdisciplinary or do they maintain a more distinct disciplinary focus toward either economics or political science?
(5) Are they featuring a narrow set of research methods or a wide variety?

Each question, examined in the context of previous research, will be addressed in turn.


Academics vs. Practitioners


Is the field of public policy relevant to practitioners? Are public policy researchers much like anthropologists who analyze another culture without intending to advise and transform it? Or are we more like the Peace Corps, offering lessons that policymakers and other practitioners can understand and perhaps implement? And do we seek to gain insights from the experience of practitioners or is their language and culture too alien?

Public policy would appear to be a field with enormous inherent relevance for practitioners. Yet, Kieser and Leiner (2012) argue that, even in applied academic fields, true collaboration between practitioners and academics is quite difficult – despite rhetorical gestures to "relevance." The contexts of science and practice are, they say, just too divergent and evidentiary standards too different. So academic researchers are sometimes characterized as unable or unwilling to contribute germane, practical insights to practitioners (Rivlin, 1984; George, 1993; deLeon, 1995; Meier & Keise, 1996; Hacker, 2010). Samuel Lewis (1993, p. ix) blamed "the two different cultures of academe and government." Policymakers who must act on "imperfect information" are frustrated by the "abstraction and jargon" of leisurely scholars who, in turn, complain practitioners put "too much faith in intuitive judgment" and reach simplistic, ad hoc conclusions.

Faced with such tension, cultural clashes, and even language gaps, have JPAM and PSJ been able to bridge this divide? We examine each journal's stated objectives, editorial board membership, publications, and authorship for evidence about whether the academic-practitioner gap has narrowed or widened in recent decades.

Stated Objectives. JPAM tells prospective contributors that, among other things, to make the submissions realistic and practical (bit.ly/jpam-a):

We seek studies that give a real-world sense of how government or non-governmental organizations operate and what they are able to do. It helps if the analysis makes use of program data or reflects contact with actual programs or agencies "on the ground."

Indeed, "government researchers and practitioners" are listed second, ahead of economists and behind political scientists, as the targeted readers of JPAM (bit.ly/jpam-p). And JPAM's listing of its invited contributor categories mentions "public managers" second (just behind economists). This is not a new ambition for JPAM. On page one of its first issue (Vernon, 1981), the editor wrote about aiming for readers who include practitioners in public agencies and in management firms. He also mentioned the goal of obtaining contributions by individuals who are "thoughtful and seasoned" practitioners (p. 3). In contrast to JPAM, PSJ has generally not been as explicit about the goal of addressing or publishing practitioners.

Editorial Boards. Editors, their editorial teams, and editorial boards ("boards" for short) frame the overall scope and content of academic journals and constitute the most engaged professionals involved their production. In 1981, JPAM began with a total of 25 board members. Fully one third were from outside universities, with five from nongovernment research organizations (e.g., Brookings), one from a foundation, one from a corporation, and one from a quasi-governmental entity in France. While none were practitioners working in government, the board still had a large non-university contingent.

By 2010, the JPAM board had expanded to 35 members, but those lacking a university affiliation had declined to three (all at think tanks), leaving the non-university minority falling from 32% to 9%. Current affiliation does not reflect lifetime associations, of course; JPAM has illustrious board members like Alice Rivlin and Burt Barnow who have rotated among academe, think tanks, and government. Nevertheless, such career shifting was the case in 1981 as well as in 2010, and the decline in non-academic membership is still substantial.

In both periods, PSJ's board was less practitioner-oriented than JPAM's but followed the same trend. In 1981, non-university representation of PSJ's board was not large (3 out of 32, 9%) but that declined (to 2 out of 55, 4%) by 2010.

Authorship. Over these nearly three decades, both JPAM and PSJ became increasingly unlikely to publish papers written by or co-authored with practitioners. Defining practitionerstrictly as anyone outside of both academia and independent research organizations, the percentage of articles with at least one practitioner author fell from 18% to 9% in JPAM and from 15% to 3% in PSJ between the early 1980s and 2007-10.

In both journals in the early 1980s, a majority of the articles with a practitioner author were sole authorships. In the later period, all but one of the dwindling number of articles by a practitioner were co-authored with at least one academic, suggesting either that practitioners could no longer get through the review process without an academic teammate or their declining interest in publishing in these journals. If author data are weighted by the number of authors per article,3 then the decline in practitioner contributionsappears in sharper relief: down from 14% to 5% in JPAM and from 13% to 1% in PSJ. During that same period there was only a slight decline in practitioner participation in the typically much shorter, less academic, and sometimes invited pieces in JPAM's various supplementary sections – from 11% to 9%.

To be sure, authors' current organizational affiliation is not a perfect measure. Some practitioners listed as affiliated with government agencies may also be former or adjunct faculty colleagues or former (or current) graduate students of their co-authors. And some current academics may have government and other "off-campus" work experience. Yet, practitioner bylines still declined.

Editors have bemoaned the difficulty in securing submissions from practitioners. One former JPAM editor commented, off the record, about the challenge of getting "papers of sufficient quality to get past even sympathetic referees" and said he had "to work very hard — soliciting, heavily editing, and guiding the manuscripts through the editorial process — to get even a few articles by practitioners." After all, he noted, practitioners are at "organizations that simply do not reward publications so the opportunity cost of publishing is very high and the rewards very low."

This cost-benefit argument helps account for the low levels of practitioner participation but not the sizeable decline. Perhaps content in both journals has become less appealing or relevant, or perhaps the peer-review hurdles have become higher. One colleague speculated that practitioners might have gone to the Public Administration Review. In fact, Raadschelders and Lee (2011) found that public administration journals had also undergone substantial declines in practitioner authorship.

Non-University Research Centers. While researchers at think tanks and other non-university research institutes may not be public managers or formal policy-makers, collectively they may be somewhat close to practitioners – frequently conducting government-funded program evaluations and occasionally advising policy-makers directly. Authorships (weighted by the number of authors) from researchers at non-university centers ebbed in PSJ from 9% to 2% and edged up slightly in JPAM from 7% to 10%, but not enough to offset the much larger decline in practitioner contributions. At the same time, in the later sections of JPAM, pages from think tank authors increased from 11% to 16%.

Management Articles. With practitioners in the audience, how much attention do these two journals give to the topic of public management? In 2011, Laurence Lynn, Jr. assembled an electronic compilation (bit.ly/jpam-l) of three decades of JPAM articles, comments, rejoinders, and short symposia pieces related to this topic. While the collection is substantively rich, especially Lynn's own JPAM contributions over the years (e.g., Lynn, 1994; and Hill & Lynn, 2004), few of the research articles came from the 2007-2010 period. In the early years, articles about management and governance, broadly construed, had constituted 14% of JPAM articles. That share fell to 6% in 2007-2010. Going beyond its research articles, JPAM's later sections show a similar pattern; management was the topic of 12% of the pages in the early years and 7% in the later years. In PSJ, research articles focusing on management and governance have held steady over time at 15-14%.


United States vs. International Coverage


In addition to the challenge of addressing both academics and practitioners is the challenge of addressing the United States and other parts of the world. Along with complaints about Anglo-American academic hegemony and the "linguistic imperialism" of English (e.g., Yeung, 2001; Paasi, 2004; Stenius, 2004; Tietze & Dick, 2013), even the Anglo side has complained about U.S. ethnocentrism. Soon after JPAM debuted, a review in a British-based journal (Kafandaris, 1982) described JPAM as "deeply North American, and quite provincial at that" (p 770).

In defense of a U.S. slant, one might note that roughly twenty European countries can be added together before matching the population and economy of the United States, making the U.S. a rather sizeable "province." At the same time, an exclusive focus on one country may deprive us of insights that might be gained by comparative analysis (Rose, 2005; Gupta 2012). Thucydides long ago saw that Greek city-states could be better analyzed comparatively rather than individually.

The challenge for a single journal is enormous and one cannot do everything. The surfeit of policy topics multiplied by nearly 200 countries makes covering all the world's public policies impossible. But do these journals at least signal that comparative analysis is important? To what extent are these two U.S.-based journals looking at public policy in other parts of the world?

Board Membership. Perhaps their editorial boards will again offer some insight into these journals. In 1981, the PSJ board had no members from outside the United States. By 2010, that had changed markedly with 11 from outside the United States (all from affluent OECD countries) – not exactly a United Nations but now 20% of the board. In contrast, JPAM began with three board members in Europe but was down to one from a non-US institution (Korea) by 2010, declining from 8% to 3% of the expanded board.

Article Focus. A large majority of attention in both journals focused on the United States and neither journal changed much over time. PSJ gave slightly more coverage to other countries than did JPAM, but despite the international additions to its board, PSJ did not show a parallel jump in international articles.4 Research articles in PSJ encompassing countries beyond the U.S. comprised 22% in the early 1980s and 23% in the later period. The corresponding share in JPAM was 17% initially and later 15%. Turning past research articles to the later sections in JPAM, during the years under study almost all of the country content was about the United States, with the exception of a two-page comparative note.

Among the articles that did go beyond the United States, both journals shifted to more comparative studies with most examining two or more countries rather than just one. This is consistent with PSJ data for the partially overlapping 2004-2009 period reported by deLeon et al. (2010). At the conclusion of that review, after noting the heavy US publication tilt but its "global circulation," PSJ editors endorsed increased "internationalization of PSJ's content" and encouraged submissions not focused exclusively on the United States as well as submissions from authors from other countries.


Individual vs. Collaborative Research


Another variable of interest in journal articles is between research by a single person and collaborative research by two or more co-authors. Editors and reviewers may increasingly prefer statistically sophisticated papers that may be more often produced by a team. Fisher et al. (1998) found that statistically advanced articles were more likely to be produced by teams of researchers in sociology, political science, and criminology. And, in the 1990s, powerful desktop computers began to support software capable of complex multivariate data analysis (Adams et al., 2013) as well as email and word processing to facilitate collaborative research to a degree never before possible. Over this period, authorship in related fields underwent a striking change. In public administration, co-authored articles ballooned from 40% in 1973 to over 85% in 2007 (Corley & Sabharwal, 2010). Similar trends occurred in economics (Maske et al., 2003), political science (Fisher et al., 1998), and sociology (Moody, 2004). What about in public policy?

Our research shows that authorship in JPAM and PSJ was transformed as well. As shown in Figure 2, co-authored articles surged in both journals from only one-third (33% in JPAM, 32% in PSJ) to about two-thirds of the articles (68% in JPAM, 64% in PSJ). The share written by two authors grew in both journals, especially in PSJ (from 28% to 41%; JPAM 26% to 34%), and articles by three or more authors also rose in both journals, especially in JPAM (7% to 34%; 4% to 23% in PSJ). Each journal even had an article with more than enough co-authors for a baseball team.5

Co-Authorship and Complex Statistics. Was the increase in co-authorship associated with increased publication of statistically complex research as Fisher et al.(1998) had found? The pattern is mixed in JPAM and nonexistent in PSJ. A large majority of JPAM articles using multivariate statistics were indeed co-authored (62% in the 1980s; 67% in the later period), far more oftenthan among those with simpler or no data analysis in the early 1980s (28% co-authored). However, in later years, among the few less quantitative JPAM articles, all eight were co-authored. In PSJ in the early 1980s, articles with multivariate statistics were somewhat more likely to be co-authored (45%) than were other articles (30%). Yet, inlater years, PSJ's multivariate articles have been slightly less likely to be co-authored (59%) than have other articles (69%). Thus, in public policy, the drive toward collaboration cannot be easily explained by the appeal of multivariate analysis for multiple authors.

Other factors, now facilitated by the electronic ease of co-authorship, may account for the enormous change. Collaborative research can offer advantages such as efficient division of labor, effective use of specialized knowledge, increased methodological depth and diversity (Laband & Tollison, 2000), increased cross-fertilization of ideas and greater intellectual camaraderie (Katz & Martin, 1997). Even the challenges, requiring compromises in design, interpretation, and presentation (Mullen & Kochan, 2001), may produce a more polished product.

Co-authorship correlates with publication quantity in diverse fields such as economics (Laband & Tolison, 2000; Maske et al., 2003), and mathematics, chemistry, and biomedicine (Glänzel, 2002), although findings are mixed for public administration (Corley & Sabharwal, 2010). Increased productivity can strengthen annual reports and promotion applications. And, at least in economics, multiple-author articles are not discounted nearly in proportion to the number of co-authors (Liner & Sewell, 2009).6





Student Co-Authors. While, as reported above, practitioners have become increasingly rare in both journals, students now appear far more often. In PSJ, the number of articles with at least one student in the byline soared from 5% to 29%, and rose in JPAM from 7% to 18%. That large increase would not have been possible without increased co-authorship with students. In PSJ, 85% of later articles with a student author were co-authored, up from 60% in the early 1980s; comparable figures for JPAM were 86%, up from 67%. Only a handful of articles were written by individual students who successfully ran the review gauntlet alone (four out of 100 later PSJ articles, and two out of 100 in the earlier PSJ period, and in each of the JPAM periods).

Are more generous faculty members now giving credit to valuable research assistants who a generation earlier would have received no more than a stipend and a footnote thanks? Whatever the dynamics, graduate students had been particular beneficiaries of the collaboration trend in both JPAM and PSJ, gaining career-enhancing publications and mentorships (Bozeman & Corely, 2004).


Economics vs. Political Science


From the beginning, scholars have grappled with public policy's messy multidisciplinary academic identity. During its emerging phase, it was often portrayed as an area that would surely meld together a vital synthesis of its parental disciplines (Lasswell, 1970) and that must do so to be successful (deLeon, 1981). But the powerful, distinctive, and valuable legacies of large,well-established disciplines like economics and political science have not easily or rapidly blended into a new unified field.

When political scientist Raymond Vernon retired as JPAM's first editor, his "swan song" essay (1985) emphasized the battles he faced forging a journal out of warring disciplines. The field of public policy, he wrote vividly, can be viewed as (p. 573-574).

…a great struggle between opposing armies. One side is labeled economists; the other side is a motley coalition bearing different banners, joined mainly by a common urge to stop the advancing economists. … [U]sing their word processors as weapons, the opposing armies pound out their hostile messages in distinctly different tongues. The actual battle, therefore, is seldom joined. Even within the coalition that is arrayed against the economists, the various members have difficulty in communicating; only when they speak of the enemy do they readily understand each other…

Most who teach, research and practice in public policy… think of their specialty as economics or law or journalism…. It is as if a faculty of clinical medicine were being drawn mainly from experts in chemistry, plumbing, butchery and statistics.

Vernon had also been frustrated facing manuscripts filled with discipline-specific terminology and words that sound ordinary but have "sharply restricted technical meaning" that others might not detect (his examples are all from economics, pp. 576-577). He also objected to the disciplinary narrowness of policy concerns(p 577):

More difficult than overcoming the problems of language has been overcoming the parochialism of authors arising out of their various specializations. Economists have dealt with the aspects of their problems that they regarded as "economic," leaving questions of politics, management, ethics, or ideology for others to consider. Political scientists, no less ethnocentric, have usually left questions of efficiency for economists.

PSJ never faced as much of a challenge of reconciling economists with non-economists since it was born with political science DNA. Stuart Nagel and others from the American Political Science Associationfounded the Policy Studies Organization in 1971. The new group launched PSJ the following year with an issue that heralded its origins with a symposium entitled "Policy Studies from a Political Science Perspective." By 1981, PSJ's editor (Dye, p. 3) was proud to note its "expansion beyond political science to economics, public administration, sociology, anthropology, and geography." But the political science link remained andin 2004 the Public Policy Section of the American Political Science Association joined the Policy Studies Organization as a co-sponsor of PSJ. In contrast, JPAM operates under the auspices of the Association for Public Policy Analysis and Management, without a formal link to a specific discipline.

The thorny question of whether public policy can ever genuinely become, as once anticipated, or ought to becomea disciplinary melting pot, is beyond the scope of this study. But we can investigate the extent to which these journals have, over nearly 30 years, become more interdisciplinary and thus may be nudging disparate sectors of the field in a common direction.

Authorship. As shown in Figure 1, although each journal drew on a somewhat different constituency, bylines (weighting by the number of authors for each article) were actually more eclectic in the early 1980s. Despite the hope of going beyond academic silos as the field matured, the opposite trend occurred as both journals became less diverse and intensified rather than reduced their disciplinary alignments.





Over time, JPAM drew increasingly on authors in economics departments or (from an economics perspective) in public policy programs and published far fewer articles by political scientists. JPAM also had relatively fewer authors identified with management and business as well as fewer practitioners. Conversely, PSJ substantially increased its publication of political scientists, along with those identified with public affairs. The only other authors to gain much representation in PSJ were from environmental programs. These increases came at the expense of non-university authors and all other academic areas, such as urban planning and sociology. PSJ had not published many economists from the outset and even that share declined.

In coding research articles, authors withjoint appointments were assigned to the first affiliation cited. And of course one could spend months further dissecting the credentials of the 705 authors of these 400 articles to calibrate their disciplinary tendencies, especially for those in research centers or interdisciplinary units such as public policy or public affairs. Our inspection of the work by authors from public policy programs, for example, reinforces the conspicuous trends shown inFigure 1; public policy authors in JPAM tended to take an economics perspective, while those in PSJ had a more political science bent.7

Citations. Another way to measure how public policy journals are situated is to examine the literature that their authors cite. What other journals are most germane to their research? What prior works ground their studies?Online ISI Journal Citation Reports were used to compile the sources cited by each journal in 2011. The top ten ISI journals cited in JPAM and PSJ articles are listed in Table 1, leaving no doubt about the segregated approaches to public policy.

Seven of the top ten journals cited in JPAM were unambiguously economics journals; none were political science journals. Of the top ten cited in PSJ, four were in political science, two in public administration, and one in public policy, plus two in overlapping areas (Political Psychology and Public Opinion Quarterly). Another notable PSJ resource was the American Economic Review, the sole journal to appear in both top ten lists.





Turning to all 2011 citations of ISI journals (excluding self-citations) indicates that JPAM's slant toward economics went beyond the top sources listed in Table 1; of all its citations, 52% were from economics journals. Just 2% of JPAM citations were from political science journals. Conversely, just 9% of PSJ citations were from economics journals and a majority was from political science (33%), public administration (13%), and public policy (10%) journals.

To elaborate on that latter category, in 2011, PSJ authors drew on research published in other public policy journals a total of 99 times, with references to JPAM, Policy Sciences, Journal of European Public Policy, Review of Policy Research, and Public Choice. JPAM authors referred to research published in other public policy journals just six times (twice to PSJ and four times to Evaluation Review).

Topics. Just as authorship and citation trends indicate divergence rather than convergence, so too do most trends in the main topics of articles. As shown in Table 2, PSJ enormously increased its attention to politics and the policy-making process (jumping from 8% to 39% of all research articles), while JPAM (also starting at 8% in early 1980s) has essentially abandoned the area. As noted earlier, JPAM also moved away from articles focusing on governance and management, while PSJ held steady in the area.





Surprisingly, both journals moved away from articles about economic policy, economic development, budget, tax, and trade policy. Likewise, articles about regulation and deregulation (a popular topic in the Reagan years) declined sharply in both journals and PSJ also discarded its early emphasis on law enforcement and crime. PSJ took up most of the slack with a large increase in articles about the policy process and politics, topics of great interest to political scientists, along with somewhat more attention to environmental and energy issues.8 JPAM especially added articles about education, welfare and poverty, the environmental and energy, and housing, as well as showing slight increases in attention to a variety of other specific policy topics.

Policy areas receiving little attention could have been consolidated into "Other" to reduce the number of rows in Table 2. Keeping them separate shows the consistently low profile of several important policy topics in these journals. It also reconfirms the ultimately impossible challenge of any general policy journal having sufficient space to publish many articles on every policy topic, no matter how consequential the topics may be. Perhaps after a topic achieves a critical mass in a journal it then attracts more submissions that help sustain the topic's presence there (e.g., recently in JPAM, education, welfare, and environmental policy; in PSJ, especially environmental policy).


Secondary Multivariate vs. Eclectic Methods


Methodological practices tend to overlap with disciplinary boundaries, but they are not synonymous. What have been the trends in the methods used in research articles in these two journals? Carlson (2011) contended that three notable policy methods trends have been toward meta-analyses, social experimentation, and Monte Carlo simulations. However it is telling that none of his 30 citations included JPAM, PSJ, or any other public policy journal for that matter. In our search, meta-analyses were rarely found and they appeared as much in the early 1980s as during 2007-2010 (three in each period in JPAM; one in each period in PSJ), no sign of a trend. Randomized, controlled trials were also rare; in 2007-2010, two were featured in PSJ and one in JPAM. Since none were found in the early 1980s, this might be generously considered a slight hint of a trend. Similarly, articles employing computer simulations, typically with Monte Carlo techniques, are also still uncommon, but increased very slightly in JPAM (3 to 5) and PSJ (0 to 1).

Little of Carlson's trends were reflected in these two journals, but what changes did we observe? Any review of methods trends must first clarify one important transformation in the content of research articles in both journals. In the early 1980s, both often published articles that looked at policy questions rather broadly, perhaps examining competing arguments and principles, drawing on illustrative data from varied sources, referencing some applicable cases. For lack of a better term, we characterize these more expansive articles as "macro" – not as synonymous with macroeconomics but as efforts to examine policy more broadly than a more narrowly focused quantitative analysis of a single dataset.

For example in JPAM, "The Bird in Hand: Feasible Strategy for Gun Control" (Moore, 1983), offered a policy analysis that assessed key objectives for gun control, detailed alternative policy options, and recommended a particular policy in light of political and other constraints. An example from PSJ is "Research Strategies for Evaluating the Adoption Potential of Energy Technologies" (Berry & Bronfman, 1981), which argued that some novel energy technology innovations are not amenable to "prospective economic analysis of adoption potential." Noneconomic models and evidence were employed such as reasoning by analogy, qualitative ratings, marketing studies, and decision-making models.

In the early 1980s, a majority of articles in both PSJ (60%) and JPAM (57%) were in this eclectic, macro category. By 2007-2010, such efforts were less common in PSJ, falling to 22%. (See .) The decline was greater in JPAM. Adding six ambitious "Policy Retrospective" articles, occasionally published in a section following research articles, puts the 2007-2010 JPAM distribution at 8% macro, 86% quantitative and 7% mixed (supplementing number crunching with some qualitative or case studies). No entirely qualitative studies were found in JPAM.





A slim majority of PSJ's articles (52%) also fell into the strict quantitative category, along with 11% that were mixed methods. PSJ also published some predominantly qualitative (15%) articles as well as a nontrivial share that fell into our macro classification (22%), yielding a substantially greater variety of approaches than in JPAM. By the 2007-2010 period, the prototypical JPAM article was likely to fit a particular formula: a secondary analysis of a large (typically government-collected) dataset using multivariate, regression-based statistics (91%). Quantitative articles in PSJ were somewhat more diverse: not so overwhelmingly reliant on secondary data (29% primary data) or on regression (33% without it). PSJ articles were also more likely to employ factor analysis or other clustering techniques (29% in PSJ vs. 2% in JPAM) and analysis of variance (27% vs. 11%).

JPAM's leadership may be rethinking its conspicuous lack of qualitative studies. In 2013, JPAM announced a forthcoming symposium on "Qualitative and Mixed-Methods for Policy Analysis" to include papers analyzing "advances in these methodologies" and evaluation research that employs these methods (bit.ly/jpam-q). Without such a bold move, JPAM's prior record would not have made it a plausible target for placing a qualitative paper.


Conclusions


The rationale for this study is that academic journals may be especially likely to influence the development of an emerging field. Early editors of public policy journals made that ambitious goal explicit. Thus, we tracked directions that two prestigious, highly competitive, public policy journals have taken since the early 1980s. This analysis of 400 research articles in JPAM and PSJ revealed a few similar patterns.

Both journals have turned increasingly "inward" to academia with fewer bylines from practitioners. The nonuniversity share of their editorial boards has also declined. Both journals have also stayed relatively "inward" geographically with the overwhelming majority of data-based articles focusing exclusively on the United States, although PSJ does have slightly more diversity in its articles and now has more international diversity on its editorial board. Also, in both journals, the proportion of co-authored articles has surged from only about one-third to about two thirds of all articles.

In one significant area, JPAM and PSJ have conspicuously diverged since the early 1980s. JPAM became far more likely to publish studies by economists (and by those in public policy programs using econometric methods) and far less likely to publish political scientists. Moving in the opposite direction, PSJ became much more likely to publish political scientists and those in public affairs programs, and less likely to publish authors from nearly all other fields except environmental studies. In this vein, JPAM authors heavily reference economics journals, rarely political science or other public policy journals. PSJ authors gravitate toward citing political science, public administration, and public policy journals, along with economics journals occasionally. These trends could be characterized as "inward" too, with a strong pull of JPAM to economics and PSJ to political science, rather than becoming increasingly interdisciplinary.

Across the same decades, JPAM moved away from broad policy analyses to a heavy focus on multivariate (essentially econometric) data analysis of large secondary data sets. PSJ also moved away from broad policy essays but presented a more diverse array of approaches, including some qualitative studies, more mixed methods, and quantitative studies using a wider variety of statistical tools.

Decisive conclusions about underlying causes are impossible without data about all papers that were submitted but not published. While we can say, for example, that "transportation policy" or "immigration policy" garnered few pages, it may be that only a handful of such articles were submitted. Nevertheless, the final patterns must reflect both the (acknowledged or unconscious) preferences of editors and reviewers along with the interests of contributing researchers – sculpted and vetted by their estimates of whether certain studies stand a chance of being published in light of a journal's recent record. Past journal performance, they may reasonably suppose, is the best available indicator of future performance. And so, without aggressive moves by editors (e.g., JPAM actively soliciting qualitative papers), journals seem more likely to evolve slowly, as they acquire a self-fulfilling reputation as especially hospitable toward certain methods and topics.

Many readers may consider it self-evident that, in the years since Raymond Vernon was at the helm, JPAM has essentially become an economics journal and that PSJ remains, while perhaps slightly more diverse than JPAM, a political science journal. But the degree and nature of that difference is of more than passing interest since it both reflects and reinforces the challenges faced by public policy in transcending long-standing academic legacies and boundaries (Brewer, 1999). The results are a far cry from early aspirations for a field that was more than just the "mingling at the frontiers" of old disciplines but rather would "integrate the various disciplines into a single movement" (Quade, 1970) and achieve the convergence Lasswell envisioned (1970).


1 Journal Citation Reports put PSJ ahead of JPAM on its "impact factor" and "immediacy" measure, while JPAM was higher on five-year impact, total cites, and "eigenfactor metrics." Focusing on the U.S., reputational ratings also elevate JPAM and PSJ as highly regarded and well known general public policy journals according to recent opinion surveys in political science (McLean et al., 2009) and public administration (Bernick & Krueger, 2010). Recent Norwegian (dbh.nsd.uib.no) and Danish rankings had the two tied in the highest category of quality journals, although two Australian rankings gave JPAM the edge (bit.ly/harzing).

2 JPAM editors were Raymond Vernon in the early period and Maureen Pirog in the 2007-2010 period. PSJ editors were Thomas Dye in 1981-85 and Hank Jenkins-Smith followed by co-editors Peter deLeon and Chris Weible in the 2007-2010 period.

3 Thus, if an article had two co-authors, they would be each weighted 0.5.

4 The Policy Studies Organization gathers an annual listing of policy scholars for Public Policy Yearbook included in a special issue of Policy Studies Journal. "Comparative public policy" was the fourth most popular area of interest, surpassed only by "governance," "environmental policy," and "social policy" (Jenkins-Smith & Trousset, 2011).

5 These ten-author articles in JPAM (King, Gakidou, Ravishankar, Moore, Lakin, Vargas, Téllez-Rojo, Ávila, Ávila, & Llamas, 2007) and PSJ (Parks, Baker, Kiser, Oakerson, Ostrom, Ostrom, Percy, Vandivort, Whitaker, & Wilson, 1981) do not rival the infamous New England Journal of Medicine article with 972 co-authors (Gusto Investigators, 1993) later surpassed in the Journal of Instrumentation with the 2,926 co-authors (Aad, et al., 2008).

6 Liner & Sewell's survey (2009) of economics chairs found only a slight discount for co-authorship when evaluating tenure applications. Holding journal reputation constant, chairs weighted productivity so that just five triple-author articles matched four single-author articles and only four and one half two-author articles equaled four single-authored articles. Co-authored articles do not necessarily yield many more subsequent citations; findings vary (cf. Corley & Sabharwal, 2010, on public administration; Barnett et al., 1988, and Piette & Ross, 1992, on economics; Nemeth & Goncalo, 2005, on psychology).

7 Authors from programs or schools with both "public policy" and "public administration" in the title were consistently not writing about public administration in these journals and were merged into the public policy umbrella.

8 These PSJ findings diverge somewhat from deLeon et al. (2010) who examined a different period (2004-2009) and tended to code articles involving the policy-making process into the topical category of the policy. We distinguished research about the policy process (grouped together regard¬less of policy issue) from policy analysis/evaluation (where the topic area was coded).


REFERENCES


  • Adams, William C., Donna L. Infeld, and Carli Wulff (2013). "Statistical Software for Curriculum and Careers." Journal of Public Affairs Education 19 (1): 173-188.
  • Aad, G., et al. (2008). "The ATLAS Experiment at the CERN Large Hadron Collider." Journal of Instrumentation 3 (8): S08003.
  • Barnett, Andy H., Richard W. Ault and David L. Kaserman. 1988. ‘The Rising Incidence of Co-authorship in Economics." The Review of Economics and Statistics 70 (3): 539–43.
  • Beed, Clive and Cara Beed. 1996. "Measuring the quality of academic journals: The case of economics." Journal of Post Keynsian Economics 18 (3): 369-396.
  • Bennett, Andrew, Aharon Barth, and Kenneth R. Rutherford. 2003. "Do we preach what we practice?" PS: Political Science and Politics 36 (3): 373-378.
  • Berry, Linda and Lois Marin Bronfman. 1981. "Research strategies for evaluating the adoption potential of energy technologies." Policy Studies Journal 9 (5): 721- 734.
  • Bernick, Ethan and Skip Krueger. 2010. "An assessment of journal quality in public administration." International Journal of Public Administration 33 (2): 98-106.
  • Biggs, Jeffrey. 2008. "Allocating the credit in collaborative research." PS: Political Science and Politics 41 (1): 246-250.
  • Bozeman, Barry and Elizabeth Corley. 2004. "Scientists' collaboration strategies." Research Policy 33 (4): 599–616.
  • Brewer, Garry D. 1974. "The policy sciences emerge: To nurture and structure a discipline." Policy Sciences 5 (3): 239-244.
  • Brewer, Garry D. 1999. "The challenges of interdisciplinary." Political Sciences 32: 327-337.
  • Corley, Elizabeth A. and Meghna Sabharwal. 2010. "Scholarly collaboration and productivity patterns in public administration." Public Administration 88 (3): 627-648.
  • deLeon, Peter. 1994. "Reinventing the policy sciences." Policy Sciences 27 (1): 77-95.
  • deLeon, Peter. 1995. "Democratic values and the policy sciences." American Journal of Political Science 39 (4): 886-905.
  • deLeon, Peter. 2010. "Editors' analysis: A status report on the Policy Studies Journal." Policy Studies Journal 38 (1): 168-173.
  • Dye, Thomas R. 1981. "Editor's notes." Policy Studies Journal 10 (1): 3-4.
  • Fisher, Bonnie S., Craig Cobane, Tomas Vander Ven and Francis T. Cullen. 1998. "How many authors does it take to publish an article?" PS: Political Science & Politics 31 (4): 847-856.
  • Frank, Robert H. 1983. "When are price differentials discriminatory?" Journal of Policy Analysis and Management 2 (2): 238-255.
  • Glänzel, Wolfgang. 2002. "Coauthorship patterns and trends in the sciences (1980–1998)." Library Trends 50 (3): 461.
  • George, Alexander L. 1993. Bridging the gap: Theory and practice in foreign policy. Washington, D.C.: United States Institute of Peace Press.
  • Gupta, Kuhink. 2012. "Comparative public policy." Policy Studies Journal 40 (S1): 11-26.
  • Gusto Investigators. 1993. "An International Randomized Trial Comparing Four Thrombolytic Strategies for Acute Myocardial Infarction." The New England Journal of Medicine 329 (10): 673-682.
  • Hacker, Jacob S. 2010. "Healing the rift between political science and practical politics." The Forum 8 (3): 1-7.
  • Heck, Jean L. and Peter A. Zaleski. 1991. ‘Trends in Economic-Journal Literature: 1969–89." Atlantic Economic Journal 19 (4): 27–32.
  • Hill, Carolyn J. and Laurence E. Lynn, Jr. 2004. "Governance and public management, an introduction." Journal of Policy Analysis and Management 23 (1): 3-11.
  • Jenkins-Smith, Hank C. and Sarah Trousset. 2011. "Public policy yearbook." Policy Studies Journal 39 (s1): vii-xiv.
  • Kafandaris, Stellos. 1982. "Review: Journal of Policy Analysis and Management." Journal of the Operational Research Society 33 (8): 770-771.
  • Katz, J. Sylvan and Ben R. Martin. 1997. ‘What Is Research Collaboration?." Research Policy 26 (1): 1–18.
  • Kieser, Alfred and Lars Leiner. 2012. "Collaborate with practitioners." Journal of Management Inquiry 21 (1): 14-28.
  • King, Gary, Emmanuela Gakidou, Nimala Ravishankar, Ryan T. Moore, Jason Lakin, Manett Vargas, Martha Maria Téllez-Rojo, Juan Eugenio Hernàndez Ávila, Mauricio Hernàndez Ávila, and Hèctor Hernández Llamas. 2007. "A ‘politically robust' experimental design for policy evaluation." Journal of Policy Analysis and Management 26 (3): 479–506.
  • Laband, David N. and Robert D. Tollison. 2000. "Intellectual collaboration." Journal of Political Economy 108 (3J): 632- 662.
  • Lasswell, Harold D. 1970. "The emerging conception of the policy sciences." Policy Sciences 1 (1): 3-14.
  • Lewis, Samuel W. 1993. Preface in George, A. L. Bridging the gap: Theory and practice in foreign policy. Washington, D.C.: United States Institute of Peace Press.
  • Liner, Gaines H. and Ellen Sewell. 2009. "Research requirements for promotion and tenure at PhD granting departments of economics." Applied Economics Letters 16: 765-768.
  • Lynn, Lawrence E., Jr. 1994. "Public management research: The triumph of art over science." Journal of Policy Analysis and Management 13 (2): 231-259.
  • Maske, Kellie L., Garey C. Durden, and Patricia E. Gaynor. 2003. "Determinants of scholarly productivity among male and female economists." Economic Inquiry 41 (4): 555-564.
  • McLean, Iain, André Blais, James C. Garad, and Michael Giles. 2009. "Comparative journal ratings: A survey report." Political Studies Review 7: 18-38.
  • Meier, Kenneth J. and Lael R. Keise. 1996. "Public administration as a science of the artificial." Public Administration Review 56 (5): 459-466.
  • Moody, James. 2004. "The structure of a social science collaboration network: Disciplinary cohesion from 1963-1999." American Sociological Review 69: 213-238.
  • Moore, Mark H. 1983. "The bird in hand: A feasible strategy for gun control." Journal of Policy Analysis and Management 2 (2): 185-195.
  • Mullen, Carol A. and Frances K. Kochan. 2001. "Issues of Collaborative Authorship in Higher Education." Educational Forum 65 (2): 128–135.
  • Nemeth, Charlan Jeanne and Jack A. Goncalo. 2005. "Creative Collaborations from Afar: The Benefits of Independent Authors' Creativity." Research Journal 17 (1): 1–8.
  • Paasi, Anssi. 2005, "Globalisation, academic capitalism, and the uneven geographies of international journal publishing spaces." Environment and Planning 37 (5): 769–789. 
  • Parks, Roger, Paula Baker, Larry Kiser, Ronald Oakerson, Elinor Ostrom, Vincent Ostrom, Stephen Percy, Martha Vandivort, Gordon Whitaker, and Rick Wilson. 1981. "Consumers as coproducers of public service." Journal of Policy Analysis and Management 9 (7):1001-1011.
  • Piette, Michael J. and Kevin L. Ross. 1992. "An Analysis of the Determinants of Co-authorship in Economics." Journal of Economic Education 23 (3): 277–283.
  • Pirog, Maureen A. 2005. "Message from the new editor." Journal of Policy Analysis and Management, 24(1): 1-4.
  • Pontille, David and Didier Torny. 2010. "The controversial polities of journal ratings." Research Evaluation 19 (5): 347-360.
  • Quade, E.S. 1970. "Why policy science?" Policy Sciences 1 (1): 1-2.
  • Raadschelders, Jos C.N. and Kwang-Hoon Lee. 2011. "Trends in the study of public administration." Public Administration Review 71 (1): 19-33.
  • Rivlin, Alice M. 1984. "Out of the Trenches: A Public Policy Paradox." Journal of Policy Analysis and Management 4 (1): 17–22.
  • Rose, Richard. 2005. Learning From Comparative Public Policy. Oxford: Routledge.
  • Scandura, Terri A. and Ethlyn A. Williams. 2000. "Research methodology in management." Academy of Management Journal 43 (6): 1248-1264.
  • Stenius, Kerstin, Isidore Obot, Florence Kerr- Corrêa, Erikson F. Furtado, and Thomas F. Barbour. Beyond the Anglo-American World. 2004. Chapter 3 in Thomas F. Babor, Kerstin Stenius, Susan Savva, and Jean O'Reilly. Publishing Addiction Science: A Guide for the Perplexed. Essex, UK: Multi-Science Publishing Corp.
  • Tietze, Susanne and Penny Dick. 2013. "The victorious English language." Journal of Management Inquiry 22 (1): 122-134.
  • Vernon, Raymond. 1981. "A word from the editor." Journal of Policy Analysis and Management 1 (1): 3-4.
  • Vernon, Raymond. 1985. "Swan song." Journal of Policy Analysis and Management 4 (4): 573-581.
  • Yeung, Henry W-C. 2001. "The two different cultures of academe and government." Environment and Planning 33: 1-9.