Guidelines on Conducting Online Experiments with Repeated Measures and Using Smartphone Applications

[Panduan Melakukan Eksperimen Daring dengan Pengukuran Multipel serta Menggunakan Aplikasi Ponsel]

  • Intan Permatasari Universitas Indonesia
  • Harry Susianto Universitas Indonesia
Abstract Views: 305 PDF - Full Text Downloads: 169
Keywords: online experiment, eksperimen daring, repeated measures, pengukuran multipel, smartphone application, aplikasi ponsel


In Indonesia, online research is typically conducted using survey design. Data collection is conducted by distributing links to questionnaires through social media. Compared to surveys, online experimental research, especially ones that utilize mobile applications and require repeated measures, are very rarely implemented. This is thought to be caused by the lack of experience among researchers of psychology in creating and managing online research platforms or smartphone applications. This article aims to guide researchers on conducting online experiments with repeated measures, integrated with additional smartphone applications that are available for free.

Di Indonesia penelitian secara daring umumnya dilakukan menggunakan desain survei. Pengumpulan datanya dilakukan dengan cara menyebarkan tautan kuesioner melalui media sosial. Dibandingkan survei, penelitian eksperimen daring, terlebih lagi yang menggunakan aplikasi ponsel dan membutuhkan pengukuran multipel sangat jarang dilakukan. Penyebabnya diduga karena peneliti psikologi umumnya belum terbiasa membuat serta mengelola platform penelitian daring atau aplikasi ponsel. Artikel ini bertujuan memberikan panduan melakukan eksperimen daring dengan pengukuran multipel (repeated measures) yang mengintegrasikan tambahan aplikasi ponsel yang bisa dilakukan secara gratis.


Download data is not yet available.


Abshire, M., Dinglas, V. D., Cajita, M. I. A., Eakin, M. N., Needham, D. M., & Himmelfarb, C. D. (2017). Participant retention practices in longitudinal clinical research studies with high retention rates. BMC Medical Research Methodology, 17:30, 1-10.

Allen, S., Graupera, V., & Lundrigan, L. (2010). The smartphone is the new PC. In Pro smartphone cross-platform development (pp. 1-14). Apress.

Amazon Mechanical Turk. (2005). Features. Amazon Mechanical Turk (MTurk).

Amazon Mechanical Turk. (2017, March 14). MTurk is now available for requesters in 10 more countries. Amazon Mechanical Turk.

American Psychological Association [APA]. (2020). Publication manual of the American Psychological Association (7th ed.).

American Psychological Association [APA]. (n.d.). Apparatus. APA Dictionary of Psychology. Retrieved August 17, 2021 from

Anjum, A., & Ilyas, M. U. (2013). Activity recognition using smartphone sensors. Proceedings of 2013 IEEE 10th Consumer Communications and Networking Conference (CCNC), 914-919.

Anwyl-Irvine, A., Dalmaijer, E. S., Hodges, N., & Evershed, J. K. (2021). Realistic precision and accuracy of online experiment platforms, web browsers, and devices. Behavior Research Methods, 53(4), 1407-1425.

Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board Task Force report. American Psychologist, 73(1), 3-25.

Atzmüller, C., & Steiner, P. M. (2010). Experimental vignette studies in survey research. Methodology, 6(3), 128-138.

Aziz, B. G. A., & Lei, S. (2016). Research on the smartphone fast growing impact on mobile gaming industry in China. Journal of Management, Marketing and Logistics, 3(4), 368-368.

Becker, R. A., Chambers, J. M., & Wilks, A. R. (1988). The new S language: A programming environment for data analysis and graphics (1st ed.). Wadsworth & Brooks/Cole Advanced Books & Software.

Boehmke, B. C. (2016). Data wrangling with R. Springer International Publishing.

Butler, J., & Kern, M. L. (2016). The PERMA-profiler: A brief multidimensional measure of flourishing. International Journal of Wellbeing, 6(3), 1-48.

Cantor, D., O’Hare, B. C., & O’Connor, K. S. (2008). The use of monetary incentives to reduce nonresponse in random digit dial telephone surveys. In J. M. Lepkowski, C. Tucker, J. M. Brick, E. D. de Leeuw, L. Japec, P. J. Lavrakas, M. W. Link, & R. L. Sangster (Eds.), Advances in telephone survey methodology (pp. 471-498). Wiley.

Capon, H., Hall, W., Fry, C., & Carter, A. (2016). Realising the technological promise of smartphones in addiction research and treatment: An ethical review. International Journal of Drug Policy, 36, 47-57.

Chan, D. (1998). The conceptualization and analysis of change over time: An integrative approach incorporating Longitudinal Mean and Covariance Structures Analysis (LMACS) and Multiple Indicator Latent Growth Modeling (MLGM). Organizational Research Methods, 1(4), 421-483.

Clough, A., Wagman, J., Rollins, C., Barnes, J., Connor-Smith, J., Holditch-Niolon, P., McDowell, S., Martinez-Bell, E., Bloom, T., Baker, C., & Glass, N. (2010). The SHARE project: Maximizing participant retention in a longitudinal study with victims of intimate partner violence. Field Methods, 23(1), 86-101.

Cook, D. A., Wittich, C. M., Daniels, W. L., West, C. P., Harris, A. M., & Beebe, T. J. (2016). Incentive and reminder strategies to improve response rate for Internet-based physician surveys: A randomized experiment. Journal of Medical Internet Research (JMIR), 18(9), 1-11.

Cotter, R. B., Burke, J. D., Loeber, R., & Navratil, J. L. (2002). Innovative retention methods in longitudinal research: A case study of the developmental trends study. Journal of Child and Family Studies, 11(4), 485-498.

den Eynden, V. V, Corti, L., Woollard, M., Bishop, L., & Horton, L. (2011). Managing and sharing data: Best practice for researchers (3rd ed.). UK Data Archive - University of Essex.

Duarte, F. (2019, September 9). Berapa banyak waktu yang dihabiskan rakyat Indonesia di media sosial? [How much time do Indonesians spend on social media?]. BBC News Indonesia.

Elliott, W. B., Hodge, F. D., & Sedor, L. M. (2012). Using online video to announce a restatement: Influences on investment decisions and the mediating role of trust. The Accounting Review, 87(2), 513-535.

Farrington, D. P. (1991). Longitudinal research strategies: Advantages, problems, and prospects. Journal of the American Academy of Child & Adolescent Psychiatry, 30(3), 369-374.

Google. (n.d.a). Show questions based on answers. Google. Retrieved March 14, 2021 from

Google. (n.d.b). Cara menggunakan Google Formulir [How to use Google Forms]. Google. Retrieved March 14, 2021 from

Google. (n.d.c). Google Formulir [Google Forms]. Google. Retrieved March 14, 2021 from

Google. (n.d.d). Choose where to save form responses. Google. Retrieved March 14, 2021 from

Gustavson, K., von Soest, T., Karevold, E., & Røysamb, E. (2012). Attrition and generalizability in longitudinal studies: Findings from a 15-year population-based study and a Monte Carlo simulation study. BMC Public Health, 12:918, 1-11.

Harari, G. M., Lane, N. D., Wang, R., Crosier, B. S., Campbell, A. T., & Gosling, S. D. (2016). Using smartphones to collect behavioral data in psychological science: Opportunities, practical considerations, and challenges. Perspectives on Psychological Science, 11(6), 838-854.

Hartman, S. J., Nelson, S. H., Cadmus-Bertram, L. A., Patterson, R. E., Parker, B. A., & Pierce, J. P. (2016). Technology- and phone-based weight loss intervention: Pilot RCT in women at elevated breast cancer risk. American Journal of Preventive Medicine, 51(5), 714-721.

Hiskey, S., & Troop, N. A. (2002). Online longitudinal survey research: Viability and participation. Social Science Computer Review Hiskey, 20(3), 250-259.

Hofer, S. M., & Hoffman, L. (2007). Statistical analysis with incomplete data: A developmental perspective. In T. D. Little, J. A. Bovaird, & N. A. Card (Eds.), Modeling contextual effects in longitudinal studies (pp. 13-32). Lawrence Erlbaum Associates Publishers.

Hooley, T., Wellens, J., & Marriott, J. (2012). What is online research? Using the internet for social science research. Bloomsbury Publishing.

Information Commissioner’s Office [ICO]. (n.d.). What are identifiers and related factors?. Information Commissioner’s Office. Retrieved March 14, 2021 from

Janke, R. (2014). Effects of mentioning the incentive prize in the email subject line on survey response. Evidence Based Library and Information Practice, 9(1), 4-13.

Jaya, E. S., Ascone, L., & Lincoln, T. M. (2018). A longitudinal mediation analysis of the effect of negative-self-schemas on positive symptoms via negative affect. Psychological Medicine, 48(8), 1299-1307.

Joint Information Systems Committee [JISC]. (2018). Research data management toolkit. Joint Information Systems Committee.

Kirono, D. C. G., & Supangat, A. (2017, December 21). Media sosial, metode riset baru ala Citizen Science. [Social media, Citizen Science's new research method.]. Mongabay: Situs Berita Lingkungan.

Kramer, I., Simons, C. J. P., Hartmann, J. A., Menne-Lothmann, C., Viechtbauer, W., Peeters, F., Schruers, K., van Bemmel, A. L., Myin-Germeys, I., & Delespaul, P. (2014). A therapeutic application of the experience sampling method in the treatment of depression: A randomized controlled trial. World Psychiatry, 13(1), 68-77.

Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004). Psychological research online: Report of Board of Scientific Affairs’ Advisory Group on the conduct of research on the internet. American Psychologist, 59(2), 105-117.

Kreuter, F., Haas, G. -C., Keusch, F., Bähr, S., & Trappmann, M. (2020). Collecting survey and smartphone sensor data with an app: Opportunities and challenges around privacy and informed consent. Social Science Computer Review, 38(5), 533-549.

Lane, N. D., Miluzzo, E., Lu, H., Peebles, D., Choudhury, T., & Campbell, A. T. (2010). A survey of mobile phone sensing. IEEE Communications Magazine, 40(9), 140-150.

Lane, N. D., Mohammod, M., Lin, M., Yang, X., Lu, H., Ali, S., Doryab, A., Berke, E., Choudhury, T., & Campbell, A. (2011, May 23-26). BeWell: A smartphone application to monitor, model and promote wellbeing. Proceedings of 5th International ICST Conference on Pervasive Computing Technologies for Healthcare, Pervasive Health, Dublin, Republic of Ireland. European Union Digital Library.

Lin, C. -C., Chang, C. -C., Liang, D., & Yang, C. -H. (2012). A new non-intrusive authentication method based on the orientation sensor for smartphone users. Proceedings of the 2012 IEEE 6th International Conference on Software Security and Reliability, 245–252.

Martin, A. A., Grünenfelder-Steiger, A. E., & Allemand, M. (2020). Longitudinal research. In B. J. Carducci, & C. S. Nave (Eds.), The Wiley encyclopedia of personality and individual differences: Measurement and assessment (pp. 25-30). Wiley.

Martinez-Martin, N., & Kreitmair, K. (2018). Ethical issues for direct-to-consumer digital psychotherapy apps: Addressing accountability, data protection, and consent. JMIR Mental Health, 5(2):e32, 1-7.

McKinney, W. (2017). Python for data analysis: Data wrangling with Pandas, Numpy, and IPython (2nd ed.). O'Reilly Media.

Miller, G. (2012). The smartphone psychology manifesto. Perspectives on Psychological Science, 7(3), 221-237.

Mourby, M., Gowans, H., Aidinlis, S., Smith, H., & Kaye, J. (2019). Governance of academic research data under the GDPR-lessons from the UK. International Data Privacy Law, 9(3), 192-206.

Musch, J., & Reips, U. D. (2000). A brief history of web experimenting. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 61-87). Elsevier.

Mychasiuk, R., & Benzies, K. (2012). Facebook: An effective tool for participant retention in longitudinal research. Child: Care, Health and Development, 38(5), 753-756.

Natalya, L. (2018). Validation of Academic Motivation Scale: Short Indonesian language version. ANIMA Indonesian Psychological Journal, 34(1), 43-53.

OpenAire. (2021). Guide for researcher: How to deal with sensitive data. OpenAire. Retrieved August 3, 2021 from

Pertiwi, W. K. (2019, January 18). WhatsApp kini lebih populer dari Facebook [WhatsApp is now more popular than Facebook]. Kompas.

Pinder, C., Vermeulen, J., & Cowan, B. R. (2018). Subliminal semantic number processing on smartphones. Proceedings of 20th International Conference on Human-Computer Interaction with Mobile Devices and Services, Spain, e24, 1-13.

Pinder, C., Vermeulen, J., Beale, R., & Hendley, R. (2015, August 25-28). Subliminal priming of nonconscious goals on smartphones [Poster session]. Proceedings of 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunc, Coppenhagen, Denmark, 825-830.

Pinder, C., Vermeulen, J., Cowan, B. R., Beale, R., & Hendley, R. J. (2017). Exploring the feasibility of subliminal priming on smartphones. Proceedings of 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, Austria, e21, 1-15.

Piwek, L., Ellis, D. A., & Andrews, S. (2016). Can programming frameworks bring smartphones into the mainstream of psychological science? Frontiers in Psychology, 7:1252.

Poon, K. -T., Teng, F., Wong, W. -Y., & Chen, Z. (2016). When nature heals: Nature exposure moderates the relationship between ostracism and aggression. Journal of Environmental Psychology, 48, 159-168.

Pouchard, L. (2015). Revisiting the data lifecycle with big data curation. International Journal of Digital Curation, 10(2), 176-192.

Purves, K. L., Constantinou, E., McGregor, T., Lester, K. J., Barry, T. J., Treanor, M., Sun, M., Margraf, J., Craske, M. G., Breen, G., & Eley, T. C. (2019). Validating the use of a smartphone app for remote administration of a fear conditioning paradigm. Behaviour Research and Therapy, 123:103475.

R Core Team. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved March 16, 2021. Available online from

Rancangan Undang-Undang Republik Indonesia Nomor … Tahun … tentang Perlindungan Data Pribadi [The Draft of Personal Data Protection Bill Number … Year … of the Republic of Indonesia]. (2019). Kementerian Komunikasi dan Informatika Republik Indonesia [Ministry of Communication and Informatics of the Republic of Indonesia]. UU PDP Final %28Setneg 061219%29.pdf

Regmi, P. R., Waithaka, E., Paudyal, A., Simkhada, P., & van Teijlingen, E. (2016). Guide to the design and application of online questionnaire surveys. Nepal Journal of Epidemiology, 6(4), 640-644.

Revilla, M., Paura, E., & Ochoa, C. (2021). Use of a research app in an online opt-in panel: The Netquest case. Methodological Innovations, 14(1).

Sarkar, T., & Roychowdhury, S. (2019). Data wrangling with Python: Creating actionable data from raw sources. Packt Publishing.

Seligman, M. (2018). PERMA and the building blocks of well-being. The Journal of Positive Psychology, 13(4), 333-335.

Shadiqi, M. A., Muluk, H., & Milla, M. N. (2018). Experiment replication: A proposed solution for developing psychological research in Indonesia. ANIMA Indonesian Psychological Journal, 33(4), 213-230.

Sianipar, A., van Groenestijn, P., & Dijkstra, T. (2016). Affective meaning, concreteness, and subjective frequency norms for Indonesian words. Frontiers in psychology, 7:1907.

Siaputra, I. B., Prawitasari, J. E., Hastjarjo, T. D., & Azwar, S. (2011). Subjective and projective measures of thesis writing procrastination: Real world and The Sims world. ANIMA Indonesian Psychological Journal, 26(2), 128-149. Available in Repositori Universitas Surabaya [Repository of University of Surabaya/UBAYA Repository] at

SmartSurvey. (n.d.). Skip logic surveys. SmartSurvey. Retrieved April 14, 2021 from

Smith, A., de Salas, K., Lewis, I., & Schüz, B. (2017). Developing smartphone apps for behavioural studies: The AlcoRisk app case study. Journal of Biomedical Informatics, 72, 108-119.

Smith, D. (2003, January). Five principles for research ethics. Monitor on Psychology, 34(1), 56.

StatCounter. (n.d.). Mobile operating system market share Indonesia. StatCounter: Global Stats. Retrieved March 14, 2021 from

Stephanie, C. (2020, November 27). Ini dia 5 penguasa pasar smartphone di Indonesia kuartal III-2020 [Here are 5 dominators of the Indonesian smartphone market in the third quarter of 2020]. Kompas.

Stockwell, K., Alabdulqader, E., Jackson, D., Basu, A., Olivier, P., & Pennington, L. (2019). Feasibility of parent communication training with remote coaching using smartphone apps. International Journal of Language and Communication Disorders, 54(2), 265-280.

Stoet, G. (2017). PsyToolkit: A novel web-based method for running online questionnaires and reaction-time experiments. Teaching of Psychology, 44(1), 24-31.

Stoycheff, E. (2016). Please participate in Part 2: Maximizing response rates in longitudinal MTurk designs. Methodological Innovations, 9.

SurveyMonkey. (n.d.a). SurveyMonkey: The world’s most popular free online survey tool. SurveyMonkey. Retrieved March 16, 2021 from

SurveyMonkey. (n.d.b). Advanced branching. SurveyMonkey. Retrieved March 16, 2021 from

SurveyMonkey. (n.d.c). Membuat riset skripsi lebih mudah dan cepat [Making undergraduate thesis research easier and faster]. SurveyMonkey. Retrieved March 16, 2021 from

SurveyMonkey. (n.d.d). Choose a plan that works for you. SurveyMonkey. Retrieved March 16, 2021 from

Susianto, H. (1992). Memadukan “ilmu” dan “seni” dalam menyusun skala Likert [Combining "science" and "art" in constructing a Likert scale]. Jurnal Psikologi Sosial, 1, 5-19.

Torous, J., Lipschitz, J., Ng, M., & Firth, J. (2020). Dropout rates in clinical trials of smartphone apps for depressive symptoms: A systematic review and meta-analysis. Journal of Affective Disorders, 263, 413-419.

UK Data Service. (n.d.). Research data management. UK Data Service. Retrieved March 16, 2021 from

Varnhagen, C. K., Gushta, M., Daniels, J., Peters, T. C., Parmar, N., Law, D., Hirsch, R., Sadler Takach, B. S., & Johnson, T. (2005). How informed is online informed consent? Ethics & Behavior, 15(1), 37-48.

Woods, K. J. P., Siegel, M. H., Traer, J., & McDermott, J. H. (2017). Headphone screening to facilitate web-based auditory experiments. Attention, Perception, & Psychophysics, 79, 2064-2072.

Yu, S., Alper, H. E., Nguyen, A. -M., Brackbill, R. M., Turner, L., Walker, D. J., Maslow, C. B., & Zweig, K. C. (2017). The effectiveness of a monetary incentive offer on survey response rates and response completeness in a longitudinal study. BMC Medical Research Methodology, 17:77.

Zakiah, N. (2021, March 9). 7 brand HP yang kuasai pasar Indonesia, banyak yang punya: Adakah merk smartphone yang kamu pakai? [7 mobile phone brands dominating the Indonesian market, many pople have it: Is the brand of smartphone you use included?]. IDN Times.

Zakirman, Z. & Rahayu, C. (2018). Popularitas WhatsApp sebagai media komunikasi dan berbagi informasi akademik mahasiswa [The popularity of WhatsApp as undergratuate students' communication and academic information sharing media]. Shaut al-Maktabah: Jurnal Perpustakaan, Arsip dan Dokumentasi, 10(1), 27-38.

How to Cite
Permatasari, I., & Susianto, H. (2022). Guidelines on Conducting Online Experiments with Repeated Measures and Using Smartphone Applications: [Panduan Melakukan Eksperimen Daring dengan Pengukuran Multipel serta Menggunakan Aplikasi Ponsel]. ANIMA Indonesian Psychological Journal, 37(1).