Advice on Getting a Start into Research
by Tao Xie
Tao Xie's Advice Portal: Also read Tao Xie's Advice on Writing Research Papers, Tools and Tips for Writing Papers, and Advice Collections, and Introductory Readings of Sofware Testing Research.
There are many ways of proceeding, depending on how much you have already read and done. Here are some general suggestions.
Guidelines on finding a research topic:
- If you don't have specific ideas of your interests (but you are interested in pursuing research in software engineering), talk to me and I will give you an overview of different directions in software engineering/testing and the directions that I am most interested in. I will give you freedom on choosing the directions that interest you most for pursuit (in other words, I won't pick a direction or problem for you). Here is a list of research directions in software engineering (but it is a bit out of dated).
- If you already have some idea of your interests (or even don't have an idea of a specific research direction but have some idea of your big research direction/topic),
After you have already decided your research topic (after discussing with me), you can do the following along the way of pursuing that topic:
- compile a bibliography on that research topic (some samples of my bibliographies are Model Checking and Testing, Requirements Verification, Analysis, and Testing). When you compile the bibliography, try to categorize them in a meaningful way. Doing so provides two benefits: you can enjoy and reuse these links throughout your research process and the process of compiling the bibliography can make you well aware the related work. Hints on how to compile the bibliography:
- search Google with keywords related to the topics (e.g., test data generation, test input generation, test generation)
- if the topic is related to testing, search through my testing researchers webpage with related keywords (e.g., object-oriented, database, architecture, UML)
- read latest papers on the topic and track down their references (especially those mentioned in the related work section). You can read Gail Kaiser's advice on finding "related work" for conference papers.
- you may find the software engineering academic genealogy helpful in finding out more related researchers in the topic: if a researcher works in your topic, his/her adviser is very likely to work in the same area, vice versa.
- Browse and keep track of proceedings of top conferences in your field and software engineering in general. You need to know what latest papers on your field are in these conferences. You can also browse and keep track of jounrals in your field and software engineering. For some papers that are not so related to your field, you don't need to read them in deailed. But once later you found out that they are relevant and you should be able to remember them and check them out. For some papers that are very relevant to your field, you should read them in a more detailed way. For a list of software engineering conferences, visit my software engineering conferences web.
- put in your Internet browser's front page the links to publications webpages of those researchers who are active in your research topic. Visit them often and keep eyes on them to see whether they have new papers out. So that you can keep updated on what they are doing and see whether you can produce new research ideas by reading their latest published papers.
- alternatively, you can use a web content update monitoring tool to save your frequent checks of the publications webpages. I use WebMon (download). Every morning, I open this software and "check all pages" to see whether there are any new updates on these webs. I don't keep the tool running all the time because it is not necessary for me to do the real time monitoring.
- sign up some mailing lists (either academic or industrial ones) that are related to your research topics. For example, if you are working in testing topics, some sample mailing lists are junit, software-testing, model-based-testing, testdrivendevelopment, at Yahoo groups. If you are using some third-party tools or libraries, you can sign up their user or discussion mailing lists, such as those of Daikon, Alloy, BCEL, and Soot.
- during the process of formulating a research problem and solutions in that research topic, you need to ask yourself the following questions (do so also when you propose an initial idea and would like to see whether it is worthwhile of investing efforts in developing/implementing the idea):
1. Is the research problem significant enough (once solved) in being practical and useful in practice? (the question is applicable only when the topic is practice-oriented; my research mostly falls into this category.)
You might have a feeling by yourself. I assume you developed software yourself in the past; then is your proposed problem really an important one based on your software development experiences? You can read Mandelin et al.'s PLDI 05 paper's first section to see how the authors came up the research problem out of their own programming experiences. To answer the question, you can also read a lot of literature or talk to different people (e.g., me, peer researchers, developers/tester in industry). The above-mentioned tip on signing up some mailing lists can help you gain a feeling by listening to people's discussion besides helping you to produce new research ideas.
2. How was the research problem addressed by the existing approaches if any? How will your proposed approach be different or better than the existing approaches (e.g., better performance, without requiring specifications, without requiring source code)?
You need to read a lot of literature and talk to different people in knowing the difference or improvement.
3. How will you evaluate your proposed approach in terms of 1). your approach addresses the research problem (in some extent); 2) your approach is better than the existing approaches if any exist?
You can see whether your evaluation can use some of the subjects for testing experiments that are collected by other researchers. In your evaluation, you can also use some open source applications, such as Daikon and Jakarta applications.
In addition, what subjects will you use in your experiments/case studies? What are the measurements you will use to reflect the performance of the approach (e.g., faster, more bugs, fewer false positives/negatives)?
Kallakuri and Elbaum's article can help you get a quick-start on doing software engineering experiments. Rothermel's class notes 18-22 on experimentations in software engineering are also useful. If you want to know more about empirical studies in general, Robert Yin's book on Case Study Research : Design and Methods is a good one. You can buy one or borrow one from me for a quick look.
The shortcut of knowing how to do a good evaluation is to read related representative papers on your topic, figure out their evaluation methodology, and try to mimic their evaluation procedure.