System of Interest

The system of interest is a statewide home-visitation-based social program Healthy Families Georgia (HFG). HFG currently operates 17 different local sites across Georgia serving at-risk first time mothers and their families within the service areas. HFG sites have screened over 10,000 first time mothers, assessed over 5,500 participants from the positively screened mothers, and served over 2,000 participants with at least one home visitation throughout Georgia since August 2003. In addition, HFG sites have provided over 44,000 home visitations to the participants throughout Georgia since August 2003. HFG sites also administer three different types of evaluation measures instruments to its participants in order to identify whether or not mothers are at-risk for child maltreatment or abuse (AAPI-2), whether or not children are developing at their normal level (ASQ), and whether or not families maintain their home environment in a proper level to raise children (HOME/SHIF).
There are approximately 180 active HFG staff members serving about 1,000 families during a typical month across Georgia. Since the Healthy Families Information System (HFGIS) launched in August 2003, Information regarding the participants has been entered in HFGIS by HFG staff members and being used to evaluate the effectiveness of the home visitation-based program, HFG, by the Center for Family Research (CFR) at the University of Georgia (UGA). Any technical reports and data processing reports that are released by CFR are shared with Healthy Families America (HFA) and other similar social programs throughout the nation as well as designated research committees.

In addition, HFGIS is the first web-based database system that is funded and developed at the state level and being successfully utilized by 100% of the agency’s operating sites in the nation. Therefore, many other state or nationwide agencies such as DFCS (GA), First Steps (GA), the Healthy Families program in other states, CDC, Healthy Families America (HFA), and Prevent Child Abuse America (PCAA) have expressed their interests in functionality, stability, reliability and effectiveness of HFGIS.
1.2 Background of the problem

Since CFR provided the initial on-site HFGIS training in February and March, 2003 and HFGIS launched in August, 2003, there has not been a concrete instructional plan or materials to train any additional HFG staff members on how to utilize HFGIS in consistent manner as well as to train existing staff members on any changes or new additions to HFGIS. It is believed due mainly to limited resources and lack of understanding and receptiveness in any alternative ways of training.

According to the Statewide HFGIS Data Report for October 1, 2003 through September 30, 2004 (Brown, Hickey & Tollett, 2004), HFG staff turnover rate during the fiscal year was 21.19% statewide with an average of 23.87% per site. As of March 2, 2005, less than 50% of current HFG staff members received the initial HFGIS training by CFR according to the HFGIS database administrator, and CFR predicts that it will be below 30% within the next a year. With such a high staff turnover rate, sites have experienced difficulties training any new staff members adequately and in timely manner.

Furthermore, there have been number of system upgrades including several overhauls in HFGIS since the HFGIS launch in August, 2003. Due to lack of proper instructional plan in place, it has been program managers’ responsibility at each site to train their staff members. As a result, staff members including program managers have struggled to understand the changes or the new additions, and become accustomed to incorporate the changes or the new additions while they use HFGIS without proper instructional plans or materials. Many staff members have misled by misunderstood program mangers, felt overwhelmed due to lack of proper instructional materials, or misinterpreted certain areas or functions, and they have gotten frustrated in turn.
1.3 Problem statement

The HFG sites and program managers are not adequately prepared for new staff members to comprehend necessary knowledge and skills on administration procedures and how to utilize HFGIS according to the guidelines. Furthermore, they are not adequately prepared for any existing staff members to adjust to changes and new additions to HFGIS and HFG policies and procedures.

Needs analysis

HFG sites have been responsible to train staff members at their sites particularly on how to administer the Evaluation Measures and how to utilize HFGIS with limited verbal guidance from HFG associates and the site liaison at UGA for the past two years. In most cases, a program manager, a clinical supervisor, and/or experienced staff members provided an overview training session followed by shadowing. Although shadowing can provide sufficient hands-on trainings in many circumstances, it has been a great concern to HFG and CFR for the quality and appropriateness of instructions due to lack of standardized instructional plans and materials.

Based on the questionnaire conducted to three HFG associates and the site liaison at UGA via email, all four responded “No” to questions “#9 - Do you think program managers are being effective training new staff on how to use HFGIS, how to administer Evaluation measures and how to collect data for Immunizations and well baby visits?” and “#10 – Do you think program managers are being effective training existing staff on new changes to HFGIS and the HFG policies and procedures?” (see Appendices A-1 and A-2). While program managers have been expected to provide the overview training of HFGIS to new staff members at their sites, it is obvious that a concrete instructional plan is much needed for both new and existing staff members along with appropriate up-to-date instructional materials to cover any on-going changes and additions to HFGIS. All four also agreed that introductory training sessions and general instructional materials are needed in the questionnaire?” (see Appendices A-1 and A-2).

Furthermore, three program managers were interviewed for this project and they consistently expressed that they do not feel confident to train new staff members without adequate instructional guidelines and materials on how to utilize HFGIS in sections that they generally do not work with. We were also able to conduct phone-interviews with seven different family support workers or family assessment workers, and they consistently stated that it would have been much easier if there were standardized instructional guidelines and materials when they came on board. During the questionnaires and phone-interviews with HFG associates, the site liaison, and the HFG staff members, they expressed positive attitudes toward using HFGIS and stated that the system is easy to use, user-friendly, helpful and useful.

In order to eliminate a question of whether or not sites are equipped with appropriate computers, printers and Internet connections to operate HFGIS in daily base, two out of three HFG associates who participated in the questionnaire confirmed that sites are equipped appropriately for the most part and one out three HFG associates said, “Not all of them” to the same question?” (see Appendices A-1 and A-2). According to the HFGIS survey conducted in January, 2004 (see Appendix A-3), 95% and 91% of HFG staff members indicated that they have computers and printers available to them at any time at work, and over 90% of HFG staff members indicated that they have fast access Internet connection available to them at any time at work. In addition, 85% of them indicated that they have proper devices available with a type of Internet connections at home.

Goal Statement

By using this instructional design, HFG sites will be able to train new HFG staff members based on their job categories and primary job functions according to the guidelines. HFG sites will also be able to provide appropriate instructions for existing HFG staff members on any changes and new additions to HFGIS in timely manner.

Upon completion of the lessons based on this instructional design, any new HFG staff will be able to enter and maintain data in HFGIS according to the guidelines, will comprehend specific knowledge related to their job performance, and review and print out any reports in HFGIS as needed. In addition, HFG staff members will be able to refer to appropriate instructional materials as needed to refresh their knowledge and skills, and they will also be able to follow certain sections of self-directed tutorials and comprehend proper knowledge and skills to adjust to any changes and new additions to HFGIS.

Rationale for the needs for instruction

The new HFG staff members will eventually learn how to perform necessary duties in HFGIS, and the existing HFG staff members will also eventually learn how to adjust to any changes and additions to HFGIS with or without the concrete instructional plans and materials as long as they have sufficient equipments to work on HFGIS and they are required to utilize HFGIS to continue their funding.

However, staff members in a site or throughout sites in HFG may get trained differently on a same issue, and it may cause confusions among the staff members in return. Program managers or clinical supervisors often times lack enough knowledge, experience, or skills to train certain sections of HFGIS due to lack of experience on the sections. As a result, some staff may get misled, misdirected, misinterpreted, and/or misunderstood.

Although sites have been somewhat effective training their new staff members to perform general duties in HFGIS, we simply do not know what they go through and how frustrated the process can be to both instructors and learners. In addition, sites experience more difficulty training existing staff members on changes or additions to the system especially when the changes are dramatically different from what they are accustomed to.

If there are a standardized instructional plan and instructional materials in place, and will be maintained up-to-date, HFG sites will be able to follow the guidelines coordinated by program managers and staff members will be able to use the instructional materials according to the guidelines. As a result, HFG staff members will receive standardized instructions throughout sites as needed in time without much confusion or frustration.
1.6 Available resources

We interviewed HFGIS site liaison as a subject matter expert in person, and conducted a questionnaire to HFG associates and the site liaison via email and follow-up phone interviews. One of the team members is the HFGIS database administrator and provided with more recent HFGIS data as needed. In addition, we used the result of HFGIS survey conducted in January, 2004, the Statewide HFGIS Data Reports (Brown, Hickey & Tollett, 2004) and the HFG Change Across Two Years Evaluation Report (Brown, Luo, Hickey & Tollett, 2005).

The degree of heterogeneity varies by site although statewide demographic information indicates it as highly heterogeneous in education level, ethnicity and age range. Based on the job categories, each HFG site consists of three groups: 1) program managers (PM) and clinical supervisors (CS) 2) family assessment workers (FAW) 3) family support workers (FSW). It is mainly because they have distinctive responsibilities within the program and, therefore, they only need to master certain portion of the program and the system in order to meet program standards except for the program managers and clinical supervisors.

We will be targeting the third group, family support workers, for this instruction because they are the main staff members who actually administer the evaluation measures and enter the data into HFGIS. In general, the older staff members get and the longer they have served, the more resistant they become to any new changes and technologies. However, they all share a positive moral of serving their state and helping at-risk families to raise their children under better environment. They may express frustrations and resistance at first, but it is mainly because they are not confident enough about a lot of new media and technologies. Therefore, it is believed that they will become much more receptive and corporative once they gain the initial confidence and become more and more comfortable working on computers and via Internet.

Prior Knowledge of Topic

In general, sites are familiar with the evaluation measures being collected by UGA since they have sent them to UGA in monthly basis from July of 2000. Sites are also familiar with utilizing HFGIS in their daily operations because it is their main operating system since August 2003. By the time they have to administer a measure and enter it into HFGIS, they should have some knowledge about the entire procedure provided by program managers, clinical supervisors or other experienced family support workers.

Entry Level Knowledge and Skills

Approximately 50% of currently active staff members received the initial HFGIS training conducted between February and March of 2003. Therefore, at least the 50% of staff members should already mastered the entry level knowledge and skills to administer and enter the evaluation measures. Nonetheless, it becomes problem with the rest of 50% of the staff members who have never received any sort of training. With its high staff turnover rate, the number of staff members who initially received the training by UGA will be reduced significantly in the near future.

For this instruction, we believe that at least 50% of target population meets the entry level knowledge and skills. In addition, we will need to conduct an entry behavior assessment prior to this instruction.

Attitudes and/or Motivation toward the Subjects

Overall, we believe that sites are motivated enough to meet their HFG standards because whether or not their fund continues depends on their performance proved by the reports that are generated by HFGIS. They are also willing to provide the better services to their participants. Therefore, we believe that they will choose to participate and corporate with anything that UGA come up with as long as they are convinced that it will in turn help them meet their requirements.

Performance Context

Managerial or Supervisor Support: HFGIS is custom designed and developed specifically for HFG funded by Children’s Trust Fund. Therefore, sites are mandated to utilize the system and encouraged to seek for further use of it. The HFG state coordinator and the technical associates are fully supportive for sites to use the system. It is also recommended by HFG to provide in-depth training for program mangers up front and then have the program managers to serve as mentors or coaches for the staff as they hire new staff members.

Physical Aspects of the Site: In order to utilize HFGIS, sites must be equipped with computers with certain specifications, proper Internet accesses and printers. Prior to August 2003, sites have funded to cover enough equipments and Internet services, and 91% of staff members stated that computers with proper Internet access are available to them at work at all times and 95% of staff members stated that printers are available to them at work at all times as of January 2004. In addition, over 90% of staff members stated that they have fast access Internet available at work at all times.

Social Aspects of the site: HFG family support workers conduct home visitations independently and evaluation measures are administered during a routine home visit. They also enter and maintain the measures independently. However, it is expected that they receive a weekly supervision by program managers or clinical supervisors so that problems and concerns should be identified and addressed in timely manner. Sites also are required to have a monthly group supervision where staff can bring their issues and concerns to everyone’s attention and have further discussion. At any time, less experienced staff members are encouraged to consult more experienced staff member including program managers, clinical supervisors and HFG technical associates with any problems, issues or concerns that they have.

Relevance of Skills to Workplace: There have been on-going constraints to the use of HFGIS for some HFG staff mainly due to lack of confidence, lack of understanding on technologies, lack of motivation to change their attitude etc. However, HFG staff members have become much more comfortable working on the system and now realize that the system is something convenient and designed to make their lives easier and not harder. Furthermore, they realize that their existing knowledge on the use of HFGIS greatly benefit them to learn any new skills when changes are made to the system.

Learning Context

HFG currently consist of 17 active sites under one HFG state coordinator and three HFG technical associates. Each HFG technical associate oversees five to six sites and conducts regular site visits to provide on-site supervision. The HFG technical associates review participant files at site visits with site staff and provide appropriate assistance as needed. It is the HFG technical associates’ responsibility to identify any problems or concerns, and consult site staff to resolve or ameliorate them. The HFG technical associates provide HFG staff training sessions which cover HFG policies and procedures, primary responsibilities, credentialing standards etc. for new staff members. It is also their responsibility to reinforce sites to follow certain procedures in proper ways as well as properly to inform sites with new changes to HFG policies and procedures or HFGIS.

Each site has a program manager who manages its daily operations including staff supervision. In addition, they are responsible to train new staff on how to administer evaluation measures and how to use HFGIS. Therefore, some sites perform better in terms of administering evaluation measures and maintaining the data in HFGIS based on the skill levels of the program manager on training and reinforcing the staff.

Some sites also have a clinical supervisor who shares supervision responsibilities on their daily operation. Their responsibilities are very similar to program managers and vary by sites.

Due to time constraint, we were only able to conduct expert review twice prior to the small-group field test, and two small-group field tests in two different HFG sites, Athens site and Gainesville site. The site liaison at UGA reviewed the lessons as an expert and two revisions were made based on the expert’s feedback prior to the small-group field tests. We sent out an email to sites regarding the formative evaluation with support from HFG to recruit voluntary participants in the beginning of the semester, and five sites expressed interests; however, we chose Athens site and Gainesville site due to locations of the sites.

Expert Review

The expert reviews were scheduled based on the expert’s time frame, and we were able to schedule the first review focusing on the content and the second review focusing on all three areas of content, technical accuracy and quality of instruction prior to the small-group field tests.

At first, the content of the first lesson was created using PowerPoint for easy review and the site liaison reviewed it to verify the information contained in the lesson. The expert was particularly interested in ensuring verbal information given in the lesson is proper and plain that learners should be able to follow it through with no further assistance.

In the second review, the lessons were completely developed and fully functional, and the expert was asked to go over the lessons in the same way as the learners would at the formative evaluations. The expert conducted all three components of the expert evaluation - content review, technical accuracy review, and quality of instruction review - as described in the Formative Evaluation Plan of this report.

Based on the expert’s feedback, revisions were made twice and development of the lessons using Captivate was completed prior to the small-group field tests.

Small-Group Field Test

Three FSW (Family Support Worker) from Athens site, and four FSW, one FAW (Family Assessment Worker) and one PM (Program manager) from Gainesville site volunteered for this formative evaluation as the learners.

We followed the Instructional Strategy Plans and the Formative Evaluation Plan as described in this report for general steps of this formative evaluation. The program of the formative evaluation (see Appendix C-2) and the lesson objectives (see Appendix C-3) were provided prior to the pre-instructional activities. The evaluators conducted the pre-instructional activities according to the Instructional Strategy Plans, and distributed an example AAPI-2 form for pre-test and a CD containing the lessons to each learner at the end of the pre-instructional activities and prior to the content presentation. After the content presentation, the learners were provided with another example of AAPI-2 form to enter it into the system as part of post-test, and asked to take the attitude questionnaire (see Appendix C-5) and the AAPI-2 formative evaluation post-test (see Appendix C-6). At the end, the learners were asked to gather and debrief in a group discussion.

The evaluators walked around the rooms while the learners participated in the content presentation at their own pace, and observed the learners’ reflections as well as assisting them and answering any questions as needed. The evaluators also coordinated the assessments including pre- and post-test, and facilitated the group discussion at last. The evaluators also suggested for the learners to conduct informal activities of the “storytelling” upon their availabilities to exchange knowledge and skills focusing on unusual cases and circumstances.

Result of the Small-Group Field Test

As listed in Table 1, the FSWs in the Athens site were 6 months, 8 months, and 2 years and 3 months from hire. All three learners stated that they had a lot of general experience with computers and Internet prior to the employment, and did not experience any problems getting used to HFGIS at all. They also believe that it is a considerable factor whether or not they have enough prior knowledge on using computers, printers and Internet in order to become employed at HFG Athens site. A learner has a master’s degree and two other learners indicated that they have some college education. They also believed that most staff hired at Athens site should already have the basic computer knowledge and skills without further instruction.

Table 1. List of learners and the lengths of services in HFG

In contrast, all learners in the Gainesville site stated that they had experienced some degree of difficulties in the beginning of their employment. Therefore, it would be helpful to have an on-line survival guide, and general information on how to operate computer, get on-line etc. Unlike that of the Athens site, they also stated that prior computer knowledge is not a factor of hiring a new staff in their site. They have been working for HFG for over two years for the program manager, four years for the FAW, and five years, four years and six months for the FSWs.

One of possible reasons for this contradiction is that all three FSWs in Athens site were Americans with at least some college education, whereas, four out of six learners in Gainesville were Hispanics with equal or less than high school completion and English as their second language.

Since the learners were somewhat experienced with administering AAPI-2 measures and entering data into HFGIS prior to the lessons, we did not find any distinctive difference entering AAPI-2 forms from pre- vs. post-test. They were consistently able to enter the forms within 5 minutes with no mistakes on both pre- and post-test. However, we expect to see some difference if the learners were new or less experienced.

As found in Figure 1, the average score on the AAPI-2 formative evaluation post-test was 83%. In Figure 1, the first three learners were from Athens site and the rest were from Gainesville site with similar 84% and 83% on average for each site. Interestingly, the lowest two scores were made by the FSWs who most interacted with the evaluators during the content presentation at each site suggesting that the interaction might have distracted the learners from comprehending the information. Knowing that the learners were experienced for approximately two years on average, we had expected for this posttest scores to be higher than 90% although it was not one of the lesson objectives. However, the only two learners out of nine scored at or above 90% were the FSW who has worked for HFG for over two years, and the program manager at the Gainesville site.

Based on the observations made by the evaluators, most of the learners used fast-forwarding function to move to one screen to another without reading the information thoroughly or comprehending the information on a screen assuming that they already have proper knowledge and skills provided in the lessons. It is believed that the learners felt comfortable entering an AAPI-2 measure and already too familiar with general information regarding AAPI-2 at the time of this evaluation. However, they did not realize that they lack some of specific knowledge since most learners fail to answer particular questions correctly. Interestingly enough, the highest scored two learners in Gainesville site despite the fact that they are not FSWs and it is not their primary job function to administer and enter AAPI-2 measures were the learners who patiently went through both lessons without skipping or fast-forwarding screens.

Figure 1. Overall result of the AAPI-2 Formative Evaluation Posttest

Another reason we believe that the learners did not pay enough attention to the lessons are there were a series of questions regarding AAPI-2 due dates which were provided in the lesson one that most learners failed to answer correctly (question #9 thru #12, see Appendix C-6), and another question in the lesson two that all the learners failed to answer correctly although the information was provided in the lesson (question #15, see Appendix C-6). In real-setting, FSWs do not need to have the skills to calculate AAPI-2 due dates by themselves since the system provides the AAPI-2 Reminder Report containing projected due dates for each of their participant in order to perform AAPI-2 administration, and therefore, they simply need to know how to print out the reminder report and follow the schedule. Nonetheless, it is important for FSWs to understand the general rules of estimating the due dates so that they are able to determine the ranges of dates that they are suppose to administer the AAPI-2 measures to a participant without fully rely on the reminder report. In the similar way, they do not necessarily need to know the definition of “Intake Date” as they were asked in the question #15 in real-setting; however, it is rather surprising to us that they have seen the term throughout the system but have not known what is meant although they may have not learned it from the lesson.

From the result of the AAPI-2 Tutorial Formative Evaluation Posttest, we generalize that the level of prior knowledge and skills on the content and educational background did not significantly influence the outcome of the lessons. Instead, level of thoroughness in reading and understanding the lessons with patience was the key to the more successful outcome after the lessons. It appears that learners with less familiarity with the content made higher degree of participation in the content presentation, and comprehended more information in return although further analyses must be conducted to verify accuracy of this outcome.

The overview of the average correction rates on the AAPI-2 posttest is provided in Appendices D-1 and D-2. Questions #1 thru #17 were related to the lesson one, and questions #18 thru #29 were related to the lesson two. On average, 77% of learners answered correctly on the lesson one related questions, and 88% of learners answered correctly on the lesson two related questions (see Appendices D-1 and D-2). We strongly believe that this result is more likely caused by the fact that the lesson one contains more verbal information and it requires memorization vs. the lesson two was performance-driven and the learners were able to follow it through without having to memorize much information. Moreover, most learners were already at the skill level that they can enter the measures in the system with no further assistance. Nevertheless, it is a significant enough difference that the style of the lesson one should be revisited and adjusted to enhance the outcome of the lesson in return (see the Recommendations for Revisions).

According to the result of the Attitude Questionnaire, FSWs currently have 13.7 cases on their caseload on average with highest of 17 and lowest of 10, and they stated that their caseloads are at least somewhat manageable. They also indicated that they administer the AAPI-2 measures mostly on-time, and Athens site have completed 100% of AAPI-2 measures that were due during January 1, 2005 thru March 31, 2005 on-time, and Gainesville site have completed 82% of AAPI-2 measures that were due during the same time period and 100% were completed on-time according to the Outcome Information Summary Report generated in HFGIS supporting their answers.

Of nine learners, five received the initial HFGIS training conducted by UGA in the spring of 2003 and four did not, and six indicated that they did not receive any type of training prior to their services at site level. In contrast, all nine answered that HFG or their site provides sufficient instruction on how to administer AAPI-2 measures and enter them in HFGIS for new staff. Since their answers were contradictory, it was hard for us to determine whether or not sites have provided sufficient enough training to new staff members. This suggests that sites do not have a consistent training plan in place to train any new staff members, and instead, they rely on hands-on training under the circumstances.

Unlike what most HFG associates and CFR expected, eight out of nine learners rated a self-directed computer-based tutorial with mixture of “In a pair with more experienced staff” and/or “In a small group of less than five staff mixed with less and more experienced staff” as the most preferred method of training which was a significant and encouraging fining for us.

In order to determine the effectiveness of the lessons, we included some questions regarding HFGIS itself so that we can distinguish certain problems that might be caused by ineffectiveness or inconvenience of the system rather than that of the instructional deign. All nine learners indicated that they are very or somewhat confident administering AAPI-2 and entering measures into HFGIS, and the evaluation section of HFGIS is easy to follow suggesting that there is no obvious problem or concern in the system that might be hampering the learners’ performance.

All nine learners said that they liked the lessons, and were very or somewhat comfortable with the computer-based setting, and the lessons were very or somewhat helpful to them. They also confirmed that the language used in the lessons was appropriate, and the user interface was very user-friendly and easy to follow. However, four learners answered, “Maybe” or “Not sure” to a question, “Would you use the tutorial and continue to refer to it if there is a residual computer-based tutorial available?”. This left us with a puzzle whether or not they truly felt the computer-based setting was effective and helpful, or whether or not they were simply being generous.

During the interviews with the learners and group discussions, learners pointed out issues that have nothing to do with HFGIS or the lessons, but influence their overall performance which in turn affect their motivations toward HFGIS and the lessons. For instance, their participants sometimes have difficulty understanding some of the questions in AAPI-2 forms although the questions should be at the 5th grade reading level. As a result, they believe that it may influence the outcome of the measures especially for Spanish participants, and AAPI-2 form layouts are different for different language versions and it is confusing for most FSWs since HFGIS follows the format of Spanish version and majority of AAPI-2 are administered using English version.

The learners also made a few suggestions for the system that might help those entering data correctly with less time and effort such as shortening the number of screens to two pages as correspond to the original AAPI-2 forms instead of four pages, adding a color on every other question line so that a question can be easily distinguishable etc.

Overall, the learners expressed that the lessons were very well designed and effective with appropriate language and user-friendly interface. Through this formative evaluation, we were encouraged that the learners are much more open and receptive to technologies and alternative ways of instruction. With a few recommendations for revisions, we believe that our instructional design will benefit the learners and bring sites with concrete instructional plans and materials that can not only be accessible through Internet but also easily be updated and maintained for long-term.

Recommendations for Revisions

First of all, since the learners who participated in the formative evaluation may have not been the ideal candidates to determine effectiveness of the lessons because all the learners have been on the job for close to or more than 6 months and the lessons were designed targeting new or less experienced staff members rather than experienced staff members, it would be ideal to conduct another series of formative evaluation consist of more than two small groups of two to six new or less experienced staff members.

Second, in order to enforce learners to pay more attention to certain parts of the lessons or motivate them to memorize certain information, we suggest to make the parts more visibly eye-catching and add pop-up “Quiz” several times during or after the lessons. By providing the quizzes, learners will be able to identify what will be the more important information to remember. Furthermore, it will be more enforcing if they must make certain level of scores to pass, and they cannot go to the next lesson unless they reach the required level in a lesson.

The third suggestion is adding an audio option to the lessons that if certain learners are not quite comfortable reading in English or not much of visual learners, they may choose to listen to the contents rather than reading them. The learners from the formative evaluation recommended this function to be optional so that if someone does not like hearing and reading at the same time, they can simply turn the audio off. We believe that this option will greatly ease the process when learners need to go over multiple lessons at a time.

Fourth, the learners suggested that they would much prefer if the lessons provided more interactivity as we expected. They stated that liked interacting with the tutorial in the beginning of the lesson two and would like to lead the procedure instead of watching what the tutorial demonstrates to them. Although we initially anticipated adding a “practice section” at the end of each lesson or at the end of related lessons, and did not have time to create the section in time for the formative evaluation, it surprised us because we did not think the learners were motivated to that level, and led us to suspect that the learners who participated in the formative evaluation were volunteered and they must be more motivated people among the target population to volunteer to spend more than two hours and efforts on the evaluation. Therefore, we suggest adding a practice section as an option until we will be able to conduct further determination on this in a subsequent formative evaluation.

Fifth, we suggest opening a dummy site allowing learners to practice their skills without worrying about affecting their caseloads in the system as a follow-up activity upon completion of the lessons. We particularly asked this to the learners during the group discussions at the end of the formative evaluation, and all nine agree that it would be a good way for them to get accustomed to the system in general and gain enough confident as a lot of people learn certain things by doing.

Lastly and most importantly, we think creating on-line group discussion boards, and/or on-line Q&A boards for learners to participate as on-going follow-up activities of the lessons. According to Glazer (2001) in the Learning Community chapter at the eBook, Learning, Teaching, and Technology by Orey (2001), “A community of practice or an educational learning community is a group of people who interact, learn together, build relationships, and in the process, develop a sense of belonging and mutual commitment.  To build a community, members must interact regularly whether face-to-face in large or small groups or online by means of e-mail, chartrooms, and teleconferencing. In fact, with the opportunity for online discussions, community members are allowed to interact more freely without any social constraints and are allowed the opportunity for additional mentoring supervision by superiors or more knowledgeable others (Greer and Hamill, 2003).  Membership in a learning community may be self-selected or assigned, but the actual level of engagement is a personal matter.  In good communities, strong bonds withstand disagreement, and members can even use conflict as a way to deepen their relationships and their learning (Wenger et al, 2002)”.

It is important to be an on-line setting so that not only participants will have access to it from any where and at any time, but also facilitators will be able to monitor and mediate discussions and provide assistance as needed in timely manner. As Buffington (2003) pointed out in his chapter, however, learners are also free to not to participate, participate but not to contribute, or contribute below the standards; as a result, this freedom can be detrimental to the individual and to the group as a whole.

In order for the on-line group discussion boards or on-line Q&A boards to be successful, HFG will need to highly encourage the learners to participate, and the learners will have to realize how helpful their experience within the discussion boards to enhance their performance. This idea may simply sound like a dream at the moment to certain managerial staff in HFG or CFR; however, we strongly recommend them to move forward on this idea although it may take some time for the learners to become familiar with using, participating, and contributing in the discussion boards on-line because if we can motivate learners enough to use this method correctly, it will be the most effective and successful ways of providing on-going follow-up activities to all staff members regardless of their job category, ethnicity, age, education level, degree of experience etc.