目录1.问题引入2.证明: getpid()返回的不是task_struct对象的pid成员单线程进程实验多线程进程实验分析getpid()tgid和pid的含义和区别3.特别提醒: pthread_create()返回的tid和内核的tid不是一回事!4.总结pid、tgid、pgid、sid的含义5.task_struct里面的group_leader6.task_struct里面的signal.thread_head结论接LKDS3.Linux内核的双向链表代码解析(3) 进程树遍历(上) 前置知识: children、sibling、real_parent、parent以及ptrace()调试进程文章继续分析linux内核的进程遍历源码涉及到线程,这里有必要说明一下PID、PGID、TID、TGID各是什么意思!注意!本文规定同一个线程组内分为主线程和新线程,新线程是由主线程创建来的,比如主线程通过pthread_create()创建1.问题引入Linux内核的getpid()系统调用可以获取调用这个函数的进程的PID,这个之前在OS18.【Linux】进程基础知识(2) PID、PPID、fork、vfork、父子进程文章讲过示例代码:#include <stdio.h> #include <unistd.h> int main() { for (;;) { printf("The father process is running! PID=%d\n",getpid()); sleep(1); } return 0; }那getpid()是让内核获取的是调用getpid()进程的task_struct对象的哪个成员的值呢?附: task_struct中和id相关的字段:struct task_struct { //...... pid_t pid; pid_t tgid; struct pid *thread_pid; //...... };如果读者之前没有研究过的话,很可能认为是pid,但其实不是!!!2.证明: getpid()返回的不是task_struct对象的pid成员这里仍然使用LKDS3.Linux内核的双向链表代码解析(3) 进程树遍历(上) 前置知识: children、sibling、real_parent、parent以及ptrace()调试进程文章的方法来调试v.7.2.2内核需要分类讨论单线程进程和多线程进程单线程进程实验使用gcc对问题引入里面的代码进行静态编译,这里我编译后的文件是test_getpid.out,将其放到虚拟机的根文件系统中,然后启动qemu+gdb调试内核启动虚拟机后:执行test_getpid.out:使用上篇文章使用的gdb命令,这里子进程的PID是73:p $lx_task_by_pid(子进程的PID)->pid p $lx_task_by_pid(子进程的PID)->tgid运行结果:结论: 对于单线程进程,该进程的task_struct对象中的pid和tgid始终相等多线程进程实验一个进程中可以有多个线程,这里使用pthread_create()来创建线程,让所有线程都调用getpid()打印pid之前在OS67.【Linux】pthread线程库的使用文章讲过,为了简单起见,主线程就不调用pthread_join()了,这里直接给代码://test_getpid.c #include <pthread.h> #include <stdio.h> #include <unistd.h> void *start_routine(void* arg) //执行这个的叫新线程 { for (;;) { printf("thread %d is running...,PID=%d\n",*(int*)arg,getpid()); sleep(1); } } int main() { pthread_t glibc_tid; int one=1,two=2,three=3; int ret1=pthread_create(&glibc_tid, NULL,start_routine, &one); int ret2=pthread_create(&glibc_tid, NULL,start_routine, &two); int ret3=pthread_create(&glibc_tid, NULL,start_routine, &three); for (;;) //执行这个的叫主线程 { printf("The process is running... PID=%d\n",getpid()); sleep(1); } return 0; }编译命令:gcc test_getpid.c -lpthread -o test_getpid.out运行结果: 主线程和新线程都打印的是同一个PIDgdb结果:之前的OS66.【Linux】初识线程和OS89.【Linux】补充资料: Linux之父对线程的理解文章就说过,Linux不区分进程和线程,而是把它们统一称为任务,即task,所以有structtask_struct这个任务控制块,每个任务里面都必须有唯一的一个任务编号,这里我简称为task_id可以看到,上面所有线程打印的PID都一样,那么getpid()返回的"pid"就不是task_id结论: getpid()返回的"pid"不能唯一标识单个任务既然getpid()返回的"pid"不是task_id,那task_struct对象里面必定有其他字段能唯一标识任务!回顾单线程进程实验结果:上图说明,单线程进程下,进程task_struct.pid==进程task_struct.tgid==73=getpid()的返回值再回顾多线程进程实验结果,可以进行合理外推,那么下面的pid和tgid都是67,应该是主线程的task_struct.pid和task_struct.tgid结论: gdb命令p $lx_task_by_pid(getppid())返回的是是主线程的task_struct对象主线程创建了许多新线程,那么内核必然要管理这些新线程,这里内核通过task_struct对象里面的thread_node来串联这些由主线程创建的新线程的task_struct对象:struct task_struct{ //...... /* PID/PID hash table linkage. */ struct pid *thread_pid; struct hlist_node pid_links[PIDTYPE_MAX]; struct list_head thread_node; //...... };注意到thread_node是struct list_head类型的,那么可以通过里面的next成员访问到所有新线程的task_struct,看看它们的pid和tgid各是什么next的值是0xffff8880046343a8,指向下一个线程的task_struct对象,换句话说,是主线程的创建的其中一个新线程,这里我称为新线程a,那么如何获取该线程的tid和tgid?答: 首先要获取到该线程的task_struct对象的首地址,有了首地址就能访问到task_struct对象里面的任何一个成员问题又来了,如何获取到该线程的task_struct对象的首地址?虽然我对gdb的命令不太熟悉,但是我可以借鉴LKDS2.Linux内核的双向链表代码解析(2) 遍历算法文章讲的container_of宏!!#define offsetof(TYPE, MEMBER) ((size_t) &((TYPE *)0)->MEMBER) #define container_of(ptr, type, member) ({ \ ((type *)(ptr - offsetof(type, member))); })先把宏全部展开:((struct task_struct *)(ptr - ((size_t) &((TYPE *)0)->MEMBER)))令ptr为0xffff8880046343a8,TYPE为struct task_struct,MEMBER为thread_node:((struct task_struct *)(0xffff8880046343a8 - ((size_t) &((struct task_struct *)0)->thread_node)))使用gdb的p命令打印就能获取新线程的task_struct对象的首地址了:p ((struct task_struct *)(0xffff8880046343a8 - ((size_t) &((struct task_struct *)0)->thread_node)))结果是0xffff888004633d00打印新线程a的pid和tgid:p ((struct task_struct *)(0xffff888004633d00))->pid p ((struct task_struct *)(0xffff888004633d00))->tgid主线程一共创建了3个新线程,下面再打印剩下的新线程的pid和tgid:新线程b:新线程c:总结为一张表格:task_struct对象首地址task_struct.pidtask_struct.tgidtask_struct.thread_node.next主线程/67670xffff8880046343a8新线程a0xffff888004633d0068670xffff888004632528新线程b0xffff888004631e8069670xffff8880046315e8新线程c0xffff888004630f407067/结论: 对于多线程的进程而言,所有线程的task_struct.tgid都一样,主线程的task_struct.pid==task_struct.tgid,主线程创建的新线程的task_struct.pid!=task_struct.tgid之前的实验所有线程调用getpid()都打印67,正好对应这里的所有线程的task_struct.tgid,所有我猜测getpid()返回是task_struct.tgid,下面看源码验证我的想法分析getpid()定义在/kernel/sys.c中:/** * sys_getpid - return the thread group id of the current process * * Note, despite the name, this returns the tgid not the pid. The tgid and * the pid are identical unless CLONE_THREAD was specified on clone() in * which case the tgid is the same in all threads of the same group. * * This is SMP safe as current->tgid does not change. */ SYSCALL_DEFINE0(getpid) { return task_tgid_vnr(current); }从上面的注释就看到了"Note, despite the name,this returns the tgid not the pid.",getpid()返回的是tgid,而不是pidtask_tgid_vnr()定义在/include/linux/pid.h中:static inline pid_t task_tgid_vnr(struct task_struct *tsk) { return __task_pid_nr_ns(tsk, PIDTYPE_TGID, NULL); }注意到PIDTYPE_TGID,这是要求内核只拿TGID,看到这里本文的目的就达到了结论: getpid()返回的是任务的task_struct.tgid,而不是task_struct.pid我猜测getpid()返回的是任务的task_struct.tgid原因是同一个线程组内,所有的线程(Linux中也称为也叫轻量级进程)的资源是共享的,这点在以下文章说过:OS66.【Linux】初识线程OS67.【Linux】pthread线程库的使用_pthread库使用OS69.【Linux】使用线程求和、pthread_self、了解clone系统调用、创建多个线程、线程栈tgid和pid的含义和区别实验做完了,根据总结的结论也有了,现在可以好好看看tgid和pid的含义和区别了,这里为了保证权威性,直接看任务调度器的维护者Ingo Molnar的回答,是在一份电子邮件中回复的Re: what's the relationship between tgid, tid and pid ?:On Thu, 5 Dec 2002, Roland McGrath wrote:> "tgid" is the PID of the whole POSIX.1 process.> "pid" is a per-thread PID-like number.there's a "PID number space" under Linux, and "PID types". Each PID typeallocates from the same base number space, but they can also share thesame PID, ie. are overlayed:enum pid_type{PIDTYPE_PID,PIDTYPE_TGID,PIDTYPE_PGID,PIDTYPE_SID,the 'TGID' is the "process PID" as in POSIX.1. The 'PID' is the same forstandalone processes and for leader threads, it's different for threads.Furthermore, the PID is also globally unique, and is recognized by thesys_tkill() interface - ie. you can send signals between threads ofdifferent "processes". The 'PGID' is the process group PID. The 'SID' isthe session ID.the kernel guarantees reference counting, ie. only when the last typedetaches a given number is it allocatable again. This is partly requiredby semantics for things like the PGID and the SID, and it's simply handyfor things like the thread identificator.Ingo我分析了一下:"The 'PID' is the same for standalone processes and forleader threads(主线程), it's different for threads." → 在standalone(单线程进程)情况下或者是多线程进程的主线程,都有task_struct.pid=task_struct.tgid"the PID is also globally unique" → PID是每个任务的唯一编号,对应内核的task_struct.pid,可以说是线程ID,即tid"the 'TGID' is the "process PID" as in POSIX.1" → TGID是POSIX.1规定的getpid()的返回值TGID的G的Group,那么TGID是线程组ID,所以为什么多线程进程实验中getpid()打印的是相同的值3.特别提醒: pthread_create()返回的tid和内核的tid不是一回事!pthread_create()返回的tid在OS70.【Linux】证明: x86_64,glibc的pthread库中,线程的tid就是线程控制块tcb的起始地址文章讲过了,pthread_create()返回的tid是线程控制块的tid,但这个线程控制块是glibc定义的,并非内核里面的换句话说: pthread_create()返回的tid属于用户态下,内核的tid(即task_struct.pid)属于内核态下 ,两种id完全不一样进而可以得出: pthread_create()创建的是线程,这个线程是库在维护的,对应在内核里面的是内核级任务 →从用户态下的库的层面来说,有线程这个概念; 从内核层面来说,没有线程这个概念,只有任务读者也可以看看理解Linux的进程，线程，PID，LWP，TID，TGID - wipan - 博客园文章给的图,分析了用户视角下和内核视角下的pid:上图很好地描述了用户视角(user view)和内核视角(kernel view)看到线程的差别：从用户视角出发，在pid 42中产生的tid 44线程，属于tgid(线程组leader的进程ID) 42。甚至用ps和top的默认参数，你都无法看到tid 44线程。从内核视角出发，tid 42和tid 44是独立的调度单元，可以把他们视为"pid 42"和"pid 44"。需要指出的是，有时候在Linux中进程和线程的区分也是不是十分严格的。即使线程和进程混用，pid和tid混用，根据上下文，还是可以清楚地区分对方想要表达的意思。上图中，从内核视角出发看到了pid 44，是从调度单元的角度出发，但是在top或ps命令中，你是绝对找不到一个pid为44的进程的，只能看到一个lwp(tid)为44的线程。4.总结pid、tgid、pgid、sid的含义之前在OS104.【Linux】守护进程(1)文章讲过pgid和sid,所以这里顺便总结pid、tgid、pgid、sid的含义为了简单起见,我就直接展示Wolfgang Mauerer所著的《Professional Linux kernel architecture》的第54页的结论5.task_struct里面的group_leader分析完了线程ID、线程组ID,顺便讲讲task_struct里面的group_leader成员仍然使用多线程进程实验的代码,调试方法一样,只是用gdb看各个线程的group_leader值上面实验说过了,getpid()打印的是task_struct.tgid,主线程的task_struct.pid==task_struct.tgid这里看看主线程的task_struct.group_leader:这里主线程的task_struct.group_leader为0xffff888004635b80,再看看0xffff888004635b80指向的task_struct对象的group_leader是多少:发现和之前一样再看看新线程a的task_struct对象的group_leader,仍然指向主线程的task_struct对象结论: 一个线程组中的所有线程的task_struct对象的group_leader指向都是该组的主线程的task_struct对象6.task_struct里面的signal.thread_headstruct task_struct的定义中包含struct signal:struct task_struct { //...... /* Signal handlers: */ struct signal_struct *signal; struct sighand_struct __rcu *sighand; //...... }struct signal_struct定义在中:/* * NOTE! "signal_struct" does not have its own * locking, because a shared signal_struct always * implies a shared sighand_struct, so locking * sighand_struct is always a proper superset of * the locking of signal_struct. */ struct signal_struct { //...... struct list_head thread_head; wait_queue_head_t wait_chldexit; /* for wait4() */ //...... /* * PR_SET_CHILD_SUBREAPER marks a process, like a service * manager, to re-parent orphan (double-forking) child processes * to this process instead of 'init'. The service manager is * able to receive SIGCHLD signals and is able to investigate * the process until it calls wait(). All children of this * process will inherit a flag if they should look for a * child_subreaper process at exit. */ unsigned int is_child_subreaper:1; unsigned int has_child_subreaper:1; unsigned int autoreap:1; //...... }所以里面的thread_head是干什么的?最好的方法是看看当年补丁贡献者提交的解释,在[PATCH 3.4 072/177] introduce for_each_thread() to replace the buggy while_each_thread()中,而且Linux创始人Linus Torvalds接纳了这个补丁!From: lizf@kernel.orgTo: stable@vger.kernel.orgCc: linux-kernel@vger.kernel.org, Oleg Nesterov <oleg@redhat.com>,"Eric W. Biederman" <ebiederm@xmission.com>,Frederic Weisbecker <fweisbec@gmail.com>,Mandeep Singh Baines <msb@chromium.org>,"Ma, Xindong" <xindong.ma@intel.com>,Michal Hocko <mhocko@suse.cz>,"Tu, Xiaobing" <xiaobing.tu@intel.com>,Andrew Morton <akpm@linux-foundation.org>,Linus Torvalds <torvalds@linux-foundation.org>,Zefan Li <lizefan@huawei.com>Subject: [PATCH 3.4 072/177] introduce for_each_thread() to replace the buggy while_each_thread()Date: Wed, 28 Jan 2015 12:08:30 +0800 [thread overview]Message-ID: <1422418236-12852-143-git-send-email-lizf@kernel.org> (raw)In-Reply-To: <1422418050-12581-1-git-send-email-lizf@kernel.org>From: Oleg Nesterov <oleg@redhat.com>3.4.106-rc1 review patch. If anyone has any objections, please let me know.------------------commit 0c740d0afc3bff0a097ad03a1c8df92757516f5c upstream.while_each_thread() and next_thread() should die, almost every locklessusage is wrong.1. Unless g == current, the lockless while_each_thread() is not safe.while_each_thread(g, t) can loop forever if g exits, next_thread()can't reach the unhashed thread in this case. Note that this canhappen even if g is the group leader, it can exec.2. Even if while_each_thread() itself was correct, people often useit wrongly.It was never safe to just take rcu_read_lock() and loop unlessyou verify that pid_alive(g) == T, even the first next_thread()can point to the already freed/reused memory.This patch addssignal_struct->thread_head and task->thread_node tocreate the normal rcu-safe list with the stable head.The newfor_each_thread(g, t) helper is always safe under rcu_read_lock() aslong as this task_struct can't go away.Note: of course it is ugly to have both task_struct->thread_node and theold task_struct->thread_group, we will kill it later, after we changethe users of while_each_thread() to use for_each_thread().Perhaps we can kill it even before we convert all users, we canreimplement next_thread(t) using the new thread_head/thread_node. Butwe can't do this right now because this will lead to subtle behaviouralchanges. For example, do/while_each_thread() always sees at least onetask, while for_each_thread() can do nothing if the whole thread grouphas died. Or thread_group_empty(), currently its semantics is not clearunless thread_group_leader(p) and we need to audit the callers before wecan change it.So this patch adds the new interface which has to coexist with the oldone for some time, hopefully the next changes will be more or lessstraightforward and the old one will go away soon.Signed-off-by: Oleg Nesterov <oleg@redhat.com>Reviewed-by: Sergey Dyasly <dserrg@gmail.com>Tested-by: Sergey Dyasly <dserrg@gmail.com>Reviewed-by: Sameer Nanda <snanda@chromium.org>Acked-by: David Rientjes <rientjes@google.com>Cc: "Eric W. Biederman" <ebiederm@xmission.com>Cc: Frederic Weisbecker <fweisbec@gmail.com>Cc: Mandeep Singh Baines <msb@chromium.org>Cc: "Ma, Xindong" <xindong.ma@intel.com>Cc: Michal Hocko <mhocko@suse.cz>Cc: "Tu, Xiaobing" <xiaobing.tu@intel.com>Signed-off-by: Andrew Morton <akpm@linux-foundation.org>Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>Signed-off-by: Zefan Li <lizefan@huawei.com>---include/linux/init_task.h | 2 ++include/linux/sched.h | 12 ++++++++++++kernel/exit.c | 1 +kernel/fork.c | 7 +++++++4 files changed, 22 insertions(+)diff --git a/include/linux/init_task.h b/include/linux/init_task.hindex e7bafa4..b11e298 100644--- a/include/linux/init_task.h+++ b/include/linux/init_task.h@@ -38,6 +38,7 @@ extern struct fs_struct init_fs;#define INIT_SIGNALS(sig) { \.nr_threads = 1, \+ .thread_head = LIST_HEAD_INIT(init_task.thread_node), \.wait_chldexit = __WAIT_QUEUE_HEAD_INITIALIZER(sig.wait_chldexit),\.shared_pending = { \.list = LIST_HEAD_INIT(sig.shared_pending.list), \@@ -202,6 +203,7 @@ extern struct task_group root_task_group;[PIDTYPE_SID] = INIT_PID_LINK(PIDTYPE_SID), \}, \.thread_group = LIST_HEAD_INIT(tsk.thread_group), \+ .thread_node = LIST_HEAD_INIT(init_signals.thread_head), \INIT_IDS \INIT_PERF_EVENTS(tsk) \INIT_TRACE_IRQFLAGS \diff --git a/include/linux/sched.h b/include/linux/sched.hindex 56d8233..d529bd9 100644--- a/include/linux/sched.h+++ b/include/linux/sched.h@@ -534,6 +534,7 @@ struct signal_struct {atomic_t sigcnt;atomic_t live;int nr_threads;+ struct list_head thread_head;wait_queue_head_t wait_chldexit; /* for wait4() */@@ -1394,6 +1395,7 @@ struct task_struct {/* PID/PID hash table linkage. */struct pid_link pids[PIDTYPE_MAX];struct list_head thread_group;+ struct list_head thread_node;struct completion *vfork_done; /* for vfork() */int __user *set_child_tid; /* CLONE_CHILD_SETTID */@@ -2397,6 +2399,16 @@ extern bool current_is_single_threaded(void);#define while_each_thread(g, t) \while ((t = next_thread(t)) != g)+#define __for_each_thread(signal, t) \+ list_for_each_entry_rcu(t, &(signal)->thread_head, thread_node)++#define for_each_thread(p, t) \+ __for_each_thread((p)->signal, t)++/* Careful: this is a double loop, 'break' won't work as expected. */+#define for_each_process_thread(p, t) \+ for_each_process(p) for_each_thread(p, t)+static inline int get_nr_threads(struct task_struct *tsk){return tsk->signal->nr_threads;diff --git a/kernel/exit.c b/kernel/exit.cindex 3eb4dcf..38980ea 100644--- a/kernel/exit.c+++ b/kernel/exit.c@@ -74,6 +74,7 @@ static void __unhash_process(struct task_struct *p, bool group_dead)__this_cpu_dec(process_counts);}list_del_rcu(&p->thread_group);+ list_del_rcu(&p->thread_node);}/*diff --git a/kernel/fork.c b/kernel/fork.cindex 0a189c0..be2495f 100644--- a/kernel/fork.c+++ b/kernel/fork.c@@ -1026,6 +1026,11 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk)sig->nr_threads = 1;atomic_set(&sig->live, 1);atomic_set(&sig->sigcnt, 1);++ /* list_add(thread_node, thread_head) without INIT_LIST_HEAD() */+ sig->thread_head = (struct list_head)LIST_HEAD_INIT(tsk->thread_node);+ tsk->thread_node = (struct list_head)LIST_HEAD_INIT(sig->thread_head);+init_waitqueue_head(&sig->wait_chldexit);if (clone_flags & CLONE_NEWPID)sig->flags |= SIGNAL_UNKILLABLE;@@ -1433,6 +1438,8 @@ static struct task_struct *copy_process(unsigned long clone_flags,p->group_leader = current->group_leader;list_add_tail_rcu(&p->thread_group,&p->group_leader->thread_group);+ list_add_tail_rcu(&p->thread_node,+ &p->signal->thread_head);}attach_pid(p, PIDTYPE_PID, pid);nr_threads++;--1.9.1但提交信息没有直白地解释signal_struct->thread_head是什么,不过不要紧,fork()创建子进程时肯定会向子进程的task_struct里面的signal_struct->thread_head设置值的,看提交者修改的代码即可,在copy_signal()中:+ /* list_add(thread_node, thread_head) without INIT_LIST_HEAD() */ + sig->thread_head = (struct list_head)LIST_HEAD_INIT(tsk->thread_node); + tsk->thread_node = (struct list_head)LIST_HEAD_INIT(sig->thread_head);tsk和sig是什么? 这就要从copy_signal()反向推出从fork()开始的调用链了!fork() └─> kernel_clone() └─> copy_process() └─> copy_signal()copy_process()定义在/kernel/fork.c中:__latent_entropy struct task_struct *copy_process( struct pid *pid, int trace, int node, struct kernel_clone_args *args) { struct task_struct *p; //...... p = dup_task_struct(current, node); //复制父进程的task_struct对象,p指向该对象,该函数的细节会在下篇文章说 //..... retval = copy_signal(clone_flags, p); if (retval) goto bad_fork_cleanup_sighand; //...... bad_fork_cleanup_sighand: __cleanup_sighand(p->sighand); //...... }这时再进入copy_signal()看看,定义在/kernel/fork.c中:static int copy_signal(u64 clone_flags, struct task_struct *tsk) { struct signal_struct *sig; if (clone_flags & CLONE_THREAD) return 0; sig = kmem_cache_zalloc(signal_cachep, GFP_KERNEL); tsk->signal = sig; if (!sig) return -ENOMEM; //...... /* list_add(thread_node, thread_head) without INIT_LIST_HEAD() */ sig->thread_head = (struct list_head)LIST_HEAD_INIT(tsk->thread_node); tsk->thread_node = (struct list_head)LIST_HEAD_INIT(sig->thread_head); //...... }给出子进程指针的传递路径:tsk指向的是创建的子进程的task_struct看到kmem_cache_zalloc(),说明sig指向的是刚创建的signal_struct对象的地址,然后把这个地址写入到tsk->signal,这样创建的signal_struct对象就属于这个子进程了接着向signal_struct对象设置值,比如sig->thread_head,设置tsk->thread_node的值:/* list_add(thread_node, thread_head) without INIT_LIST_HEAD() */ sig->thread_head = (struct list_head)LIST_HEAD_INIT(tsk->thread_node); tsk->thread_node = (struct list_head)LIST_HEAD_INIT(sig->thread_head);注: LIST_HEAD_INIT之前在Linux内核的双向链表代码解析(1) 初始化、插入、删除文章讲过展开后:sig->thread_head = (struct list_head){ &(tsk->thread_node), &(tsk->thread_node) } tsk->thread_node = (struct list_head){ &(sig->thread_head), &(sig->thread_head) }那么有:(sig->thread_head).next==&(tsk->thread_node) (sig->thread_head).prev==&(tsk->thread_node) (tsk->thread_node).next==&(sig->thread_head) (tsk->thread_node).prev==&(sig->thread_head)画个图就很容易理解,设置前:设置后,是一个“自循环”的链表:那如果这个刚刚创建子进程创建自己的第一个新线程,上面的图会怎么变化呢?答: 通过pthread_create()创建线程,copy_signal会直接返回,不分配新的signal_struct!!!static int copy_signal(u64 clone_flags, struct task_struct *tsk) { struct signal_struct *sig; if (clone_flags & CLONE_THREAD) //如果是线程,那么不会分配signal_struct,直接返回 return 0; //...... }CLONE_THREAD宏定义在/include/uapi/linux/sched.h:#define CLONE_THREAD 0x00010000 /* Same thread group? */结论: 同一个线程组内,signal_struct只会创建一次那其他线程的task_struct的struct signal_struct* signal指向谁呢?答: 我猜测所有线程共享同一个signal,这里可以调试内核验证,这里为了简单起见,稍微修改一下多线程进程实验的代码://test_thread_head.c #include <pthread.h> #include <stdio.h> #include <unistd.h> void *start_routine(void* arg) //执行这个的叫新线程 { for (;;) { printf("thread %d is running...,PID=%d\n",*(int*)arg,getpid()); sleep(1); } } int main() { pthread_t glibc_tid; int one=1; int ret=pthread_create(&glibc_tid, NULL,start_routine, &one); for (;;) //执行这个的叫主线程 { printf("The process is running... PID=%d,shell's PID=%d\n",getpid(),getppid()); sleep(1); } return 0; }gcc -static test_thread_head.c -o test_thread_head.out再调试内核看看:执行test_thread_head.out,那么shell就fork()+execve()出了test_thread_head.out进程,这个进程再创建一个线程gdb调试时要注意看所有线程的task_struct.signal是不是指向同一个对象,即看看signal指针的值是否一样:发现主线程和新线程输出的的signal值一样,都为0xffff8880044eb180,说明指向的是同一个signal_struct对象!结论: 同一个线程组内,所有线程共享同一个signal_struct对象.体现在所有线程的task_struct.signal的指针的值相同不要退出调试,这时再看看新线程的thread_node的next指向谁:发现next值为0xffff8880044eb190,恰好等于主线程的thread_node对象里面的prev的值(注意,这里说的是两个指针变量本身存储的值是一样的)那0xffff8880044eb190代表什么? 结合之前的画的图,这个值很有可能指向共享的signal_struct的thread_head的成员!使用gdb验证,算出来的值为0xffff8880044eb180,确实是共享的signal_struct对象给出所有的gdb命令和结果,方便读者复制后调试:(gdb) p $lx_task_by_pid(66)->signal $1 = (struct signal_struct *) 0xffff8880044eb180 (gdb) p $lx_task_by_pid(66)->thread_node $2 = {next = 0xffff8880046343a8, prev = 0xffff8880044eb190} (gdb) p ((struct task_struct *)(0xffff8880046343a8 - ((size_t) &((struct task_struct *)0)->thread_node))) $3 = (struct task_struct *) 0xffff888004633d00 (gdb) p ((struct task_struct *)(0xffff888004633d00))->signal $4 = (struct signal_struct *) 0xffff8880044eb180 (gdb) p ((struct task_struct *)(0xffff888004633d00))->thread_node $5 = {next = 0xffff8880044eb190, prev = 0xffff888004637168} (gdb) p ((struct signal_struct *)(0xffff8880044eb190 - ((size_t) &((struct signal_struct *)0)->thread_head))) $6 = (struct signal_struct *) 0xffff8880044eb180可以画创建一个新线程后的图了! 如下:上面的图比较复杂,箭头太多了,可以把上面的图拆开为2个图去理解,如下:图1: signal_struct是所有线程共享的同一个 signal_struct signal_struct (thread_head, nr_threads, ...) ▲ ▲ ▲ ▲ │ │ │ │ signal│ signal│ signal│ signal│ │ │ │ │ 组长.task 新线程1.task 新线程2.task ... 最后线程.task thread_node thread_node thread_node thread_node图2: 线程组内所有线程之间的组织情况附memraid代码,然后放到Free Mermaid Editor Online | PNG, SVG & PDF | Mermaid Live Editor里面生图flowchart TD subgraph SIG["signal_struct"] TH["signal->thread_head (哨兵)"] end subgraph TS0["task_struct (主线程/组长)"] T0["主线程 (组长).thread_node"] end subgraph TS1["task_struct (新线程1)"] T1["新线程1.thread_node"] end DOTS["..."] subgraph TSn["task_struct (新线程n)"] Tn["新线程n.thread_node"] end TH <--> T0 T0 <--> T1 T1 <-->DOTS DOTS<-->Tn Tn <--> TH结论同一个线程组内,所有线程的 task_struct.signal都共享同一个signal_struct对象,该对象是主线程创建时分配空间的,主线程创建的线程不分配signal_struct对象thread_head.next指向主线程的thread_node,thread_head.prev指向最后一个线程的 thread_node,通过thread_head就能遍历所有同一个线程组内的所有线程
目录1.问题引入2.证明: getpid()返回的不是task_struct对象的pid成员单线程进程实验多线程进程实验分析getpid()tgid和pid的含义和区别3.特别提醒: pthread_create()返回的tid和内核的tid不是一回事!4.总结pid、tgid、pgid、sid的含义5.task_struct里面的group_leader6.task_struct里面的signal.thread_head结论接LKDS3.Linux内核的双向链表代码解析(3) 进程树遍历(上) 前置知识: children、sibling、real_parent、parent以及ptrace()调试进程文章继续分析linux内核的进程遍历源码涉及到线程,这里有必要说明一下PID、PGID、TID、TGID各是什么意思!注意!本文规定同一个线程组内分为主线程和新线程,新线程是由主线程创建来的,比如主线程通过pthread_create()创建1.问题引入Linux内核的getpid()系统调用可以获取调用这个函数的进程的PID,这个之前在OS18.【Linux】进程基础知识(2) PID、PPID、fork、vfork、父子进程文章讲过示例代码:#include <stdio.h> #include <unistd.h> int main() { for (;;) { printf("The father process is running! PID=%d\n",getpid()); sleep(1); } return 0; }那getpid()是让内核获取的是调用getpid()进程的task_struct对象的哪个成员的值呢?附: task_struct中和id相关的字段:struct task_struct { //...... pid_t pid; pid_t tgid; struct pid *thread_pid; //...... };如果读者之前没有研究过的话,很可能认为是pid,但其实不是!!!2.证明: getpid()返回的不是task_struct对象的pid成员这里仍然使用LKDS3.Linux内核的双向链表代码解析(3) 进程树遍历(上) 前置知识: children、sibling、real_parent、parent以及ptrace()调试进程文章的方法来调试v.7.2.2内核需要分类讨论单线程进程和多线程进程单线程进程实验使用gcc对问题引入里面的代码进行静态编译,这里我编译后的文件是test_getpid.out,将其放到虚拟机的根文件系统中,然后启动qemu+gdb调试内核启动虚拟机后:执行test_getpid.out:使用上篇文章使用的gdb命令,这里子进程的PID是73:p $lx_task_by_pid(子进程的PID)->pid p $lx_task_by_pid(子进程的PID)->tgid运行结果:结论: 对于单线程进程,该进程的task_struct对象中的pid和tgid始终相等多线程进程实验一个进程中可以有多个线程,这里使用pthread_create()来创建线程,让所有线程都调用getpid()打印pid之前在OS67.【Linux】pthread线程库的使用文章讲过,为了简单起见,主线程就不调用pthread_join()了,这里直接给代码://test_getpid.c #include <pthread.h> #include <stdio.h> #include <unistd.h> void *start_routine(void* arg) //执行这个的叫新线程 { for (;;) { printf("thread %d is running...,PID=%d\n",*(int*)arg,getpid()); sleep(1); } } int main() { pthread_t glibc_tid; int one=1,two=2,three=3; int ret1=pthread_create(&glibc_tid, NULL,start_routine, &one); int ret2=pthread_create(&glibc_tid, NULL,start_routine, &two); int ret3=pthread_create(&glibc_tid, NULL,start_routine, &three); for (;;) //执行这个的叫主线程 { printf("The process is running... PID=%d\n",getpid()); sleep(1); } return 0; }编译命令:gcc test_getpid.c -lpthread -o test_getpid.out运行结果: 主线程和新线程都打印的是同一个PIDgdb结果:之前的OS66.【Linux】初识线程和OS89.【Linux】补充资料: Linux之父对线程的理解文章就说过,Linux不区分进程和线程,而是把它们统一称为任务,即task,所以有structtask_struct这个任务控制块,每个任务里面都必须有唯一的一个任务编号,这里我简称为task_id可以看到,上面所有线程打印的PID都一样,那么getpid()返回的"pid"就不是task_id结论: getpid()返回的"pid"不能唯一标识单个任务既然getpid()返回的"pid"不是task_id,那task_struct对象里面必定有其他字段能唯一标识任务!回顾单线程进程实验结果:上图说明,单线程进程下,进程task_struct.pid==进程task_struct.tgid==73=getpid()的返回值再回顾多线程进程实验结果,可以进行合理外推,那么下面的pid和tgid都是67,应该是主线程的task_struct.pid和task_struct.tgid结论: gdb命令p $lx_task_by_pid(getppid())返回的是是主线程的task_struct对象主线程创建了许多新线程,那么内核必然要管理这些新线程,这里内核通过task_struct对象里面的thread_node来串联这些由主线程创建的新线程的task_struct对象:struct task_struct{ //...... /* PID/PID hash table linkage. */ struct pid *thread_pid; struct hlist_node pid_links[PIDTYPE_MAX]; struct list_head thread_node; //...... };注意到thread_node是struct list_head类型的,那么可以通过里面的next成员访问到所有新线程的task_struct,看看它们的pid和tgid各是什么next的值是0xffff8880046343a8,指向下一个线程的task_struct对象,换句话说,是主线程的创建的其中一个新线程,这里我称为新线程a,那么如何获取该线程的tid和tgid?答: 首先要获取到该线程的task_struct对象的首地址,有了首地址就能访问到task_struct对象里面的任何一个成员问题又来了,如何获取到该线程的task_struct对象的首地址?虽然我对gdb的命令不太熟悉,但是我可以借鉴LKDS2.Linux内核的双向链表代码解析(2) 遍历算法文章讲的container_of宏!!#define offsetof(TYPE, MEMBER) ((size_t) &((TYPE *)0)->MEMBER) #define container_of(ptr, type, member) ({ \ ((type *)(ptr - offsetof(type, member))); })先把宏全部展开:((struct task_struct *)(ptr - ((size_t) &((TYPE *)0)->MEMBER)))令ptr为0xffff8880046343a8,TYPE为struct task_struct,MEMBER为thread_node:((struct task_struct *)(0xffff8880046343a8 - ((size_t) &((struct task_struct *)0)->thread_node)))使用gdb的p命令打印就能获取新线程的task_struct对象的首地址了:p ((struct task_struct *)(0xffff8880046343a8 - ((size_t) &((struct task_struct *)0)->thread_node)))结果是0xffff888004633d00打印新线程a的pid和tgid:p ((struct task_struct *)(0xffff888004633d00))->pid p ((struct task_struct *)(0xffff888004633d00))->tgid主线程一共创建了3个新线程,下面再打印剩下的新线程的pid和tgid:新线程b:新线程c:总结为一张表格:task_struct对象首地址task_struct.pidtask_struct.tgidtask_struct.thread_node.next主线程/67670xffff8880046343a8新线程a0xffff888004633d0068670xffff888004632528新线程b0xffff888004631e8069670xffff8880046315e8新线程c0xffff888004630f407067/结论: 对于多线程的进程而言,所有线程的task_struct.tgid都一样,主线程的task_struct.pid==task_struct.tgid,主线程创建的新线程的task_struct.pid!=task_struct.tgid之前的实验所有线程调用getpid()都打印67,正好对应这里的所有线程的task_struct.tgid,所有我猜测getpid()返回是task_struct.tgid,下面看源码验证我的想法分析getpid()定义在/kernel/sys.c中:/** * sys_getpid - return the thread group id of the current process * * Note, despite the name, this returns the tgid not the pid. The tgid and * the pid are identical unless CLONE_THREAD was specified on clone() in * which case the tgid is the same in all threads of the same group. * * This is SMP safe as current->tgid does not change. */ SYSCALL_DEFINE0(getpid) { return task_tgid_vnr(current); }从上面的注释就看到了"Note, despite the name,this returns the tgid not the pid.",getpid()返回的是tgid,而不是pidtask_tgid_vnr()定义在/include/linux/pid.h中:static inline pid_t task_tgid_vnr(struct task_struct *tsk) { return __task_pid_nr_ns(tsk, PIDTYPE_TGID, NULL); }注意到PIDTYPE_TGID,这是要求内核只拿TGID,看到这里本文的目的就达到了结论: getpid()返回的是任务的task_struct.tgid,而不是task_struct.pid我猜测getpid()返回的是任务的task_struct.tgid原因是同一个线程组内,所有的线程(Linux中也称为也叫轻量级进程)的资源是共享的,这点在以下文章说过:OS66.【Linux】初识线程OS67.【Linux】pthread线程库的使用_pthread库使用OS69.【Linux】使用线程求和、pthread_self、了解clone系统调用、创建多个线程、线程栈tgid和pid的含义和区别实验做完了,根据总结的结论也有了,现在可以好好看看tgid和pid的含义和区别了,这里为了保证权威性,直接看任务调度器的维护者Ingo Molnar的回答,是在一份电子邮件中回复的Re: what's the relationship between tgid, tid and pid ?:On Thu, 5 Dec 2002, Roland McGrath wrote:> "tgid" is the PID of the whole POSIX.1 process.> "pid" is a per-thread PID-like number.there's a "PID number space" under Linux, and "PID types". Each PID typeallocates from the same base number space, but they can also share thesame PID, ie. are overlayed:enum pid_type{PIDTYPE_PID,PIDTYPE_TGID,PIDTYPE_PGID,PIDTYPE_SID,the 'TGID' is the "process PID" as in POSIX.1. The 'PID' is the same forstandalone processes and for leader threads, it's different for threads.Furthermore, the PID is also globally unique, and is recognized by thesys_tkill() interface - ie. you can send signals between threads ofdifferent "processes". The 'PGID' is the process group PID. The 'SID' isthe session ID.the kernel guarantees reference counting, ie. only when the last typedetaches a given number is it allocatable again. This is partly requiredby semantics for things like the PGID and the SID, and it's simply handyfor things like the thread identificator.Ingo我分析了一下:"The 'PID' is the same for standalone processes and forleader threads(主线程), it's different for threads." → 在standalone(单线程进程)情况下或者是多线程进程的主线程,都有task_struct.pid=task_struct.tgid"the PID is also globally unique" → PID是每个任务的唯一编号,对应内核的task_struct.pid,可以说是线程ID,即tid"the 'TGID' is the "process PID" as in POSIX.1" → TGID是POSIX.1规定的getpid()的返回值TGID的G的Group,那么TGID是线程组ID,所以为什么多线程进程实验中getpid()打印的是相同的值3.特别提醒: pthread_create()返回的tid和内核的tid不是一回事!pthread_create()返回的tid在OS70.【Linux】证明: x86_64,glibc的pthread库中,线程的tid就是线程控制块tcb的起始地址文章讲过了,pthread_create()返回的tid是线程控制块的tid,但这个线程控制块是glibc定义的,并非内核里面的换句话说: pthread_create()返回的tid属于用户态下,内核的tid(即task_struct.pid)属于内核态下 ,两种id完全不一样进而可以得出: pthread_create()创建的是线程,这个线程是库在维护的,对应在内核里面的是内核级任务 →从用户态下的库的层面来说,有线程这个概念; 从内核层面来说,没有线程这个概念,只有任务读者也可以看看理解Linux的进程，线程，PID，LWP，TID，TGID - wipan - 博客园文章给的图,分析了用户视角下和内核视角下的pid:上图很好地描述了用户视角(user view)和内核视角(kernel view)看到线程的差别：从用户视角出发，在pid 42中产生的tid 44线程，属于tgid(线程组leader的进程ID) 42。甚至用ps和top的默认参数，你都无法看到tid 44线程。从内核视角出发，tid 42和tid 44是独立的调度单元，可以把他们视为"pid 42"和"pid 44"。需要指出的是，有时候在Linux中进程和线程的区分也是不是十分严格的。即使线程和进程混用，pid和tid混用，根据上下文，还是可以清楚地区分对方想要表达的意思。上图中，从内核视角出发看到了pid 44，是从调度单元的角度出发，但是在top或ps命令中，你是绝对找不到一个pid为44的进程的，只能看到一个lwp(tid)为44的线程。4.总结pid、tgid、pgid、sid的含义之前在OS104.【Linux】守护进程(1)文章讲过pgid和sid,所以这里顺便总结pid、tgid、pgid、sid的含义为了简单起见,我就直接展示Wolfgang Mauerer所著的《Professional Linux kernel architecture》的第54页的结论5.task_struct里面的group_leader分析完了线程ID、线程组ID,顺便讲讲task_struct里面的group_leader成员仍然使用多线程进程实验的代码,调试方法一样,只是用gdb看各个线程的group_leader值上面实验说过了,getpid()打印的是task_struct.tgid,主线程的task_struct.pid==task_struct.tgid这里看看主线程的task_struct.group_leader:这里主线程的task_struct.group_leader为0xffff888004635b80,再看看0xffff888004635b80指向的task_struct对象的group_leader是多少:发现和之前一样再看看新线程a的task_struct对象的group_leader,仍然指向主线程的task_struct对象结论: 一个线程组中的所有线程的task_struct对象的group_leader指向都是该组的主线程的task_struct对象6.task_struct里面的signal.thread_headstruct task_struct的定义中包含struct signal:struct task_struct { //...... /* Signal handlers: */ struct signal_struct *signal; struct sighand_struct __rcu *sighand; //...... }struct signal_struct定义在中:/* * NOTE! "signal_struct" does not have its own * locking, because a shared signal_struct always * implies a shared sighand_struct, so locking * sighand_struct is always a proper superset of * the locking of signal_struct. */ struct signal_struct { //...... struct list_head thread_head; wait_queue_head_t wait_chldexit; /* for wait4() */ //...... /* * PR_SET_CHILD_SUBREAPER marks a process, like a service * manager, to re-parent orphan (double-forking) child processes * to this process instead of 'init'. The service manager is * able to receive SIGCHLD signals and is able to investigate * the process until it calls wait(). All children of this * process will inherit a flag if they should look for a * child_subreaper process at exit. */ unsigned int is_child_subreaper:1; unsigned int has_child_subreaper:1; unsigned int autoreap:1; //...... }所以里面的thread_head是干什么的?最好的方法是看看当年补丁贡献者提交的解释,在[PATCH 3.4 072/177] introduce for_each_thread() to replace the buggy while_each_thread()中,而且Linux创始人Linus Torvalds接纳了这个补丁!From: lizf@kernel.orgTo: stable@vger.kernel.orgCc: linux-kernel@vger.kernel.org, Oleg Nesterov <oleg@redhat.com>,"Eric W. Biederman" <ebiederm@xmission.com>,Frederic Weisbecker <fweisbec@gmail.com>,Mandeep Singh Baines <msb@chromium.org>,"Ma, Xindong" <xindong.ma@intel.com>,Michal Hocko <mhocko@suse.cz>,"Tu, Xiaobing" <xiaobing.tu@intel.com>,Andrew Morton <akpm@linux-foundation.org>,Linus Torvalds <torvalds@linux-foundation.org>,Zefan Li <lizefan@huawei.com>Subject: [PATCH 3.4 072/177] introduce for_each_thread() to replace the buggy while_each_thread()Date: Wed, 28 Jan 2015 12:08:30 +0800 [thread overview]Message-ID: <1422418236-12852-143-git-send-email-lizf@kernel.org> (raw)In-Reply-To: <1422418050-12581-1-git-send-email-lizf@kernel.org>From: Oleg Nesterov <oleg@redhat.com>3.4.106-rc1 review patch. If anyone has any objections, please let me know.------------------commit 0c740d0afc3bff0a097ad03a1c8df92757516f5c upstream.while_each_thread() and next_thread() should die, almost every locklessusage is wrong.1. Unless g == current, the lockless while_each_thread() is not safe.while_each_thread(g, t) can loop forever if g exits, next_thread()can't reach the unhashed thread in this case. Note that this canhappen even if g is the group leader, it can exec.2. Even if while_each_thread() itself was correct, people often useit wrongly.It was never safe to just take rcu_read_lock() and loop unlessyou verify that pid_alive(g) == T, even the first next_thread()can point to the already freed/reused memory.This patch addssignal_struct->thread_head and task->thread_node tocreate the normal rcu-safe list with the stable head.The newfor_each_thread(g, t) helper is always safe under rcu_read_lock() aslong as this task_struct can't go away.Note: of course it is ugly to have both task_struct->thread_node and theold task_struct->thread_group, we will kill it later, after we changethe users of while_each_thread() to use for_each_thread().Perhaps we can kill it even before we convert all users, we canreimplement next_thread(t) using the new thread_head/thread_node. Butwe can't do this right now because this will lead to subtle behaviouralchanges. For example, do/while_each_thread() always sees at least onetask, while for_each_thread() can do nothing if the whole thread grouphas died. Or thread_group_empty(), currently its semantics is not clearunless thread_group_leader(p) and we need to audit the callers before wecan change it.So this patch adds the new interface which has to coexist with the oldone for some time, hopefully the next changes will be more or lessstraightforward and the old one will go away soon.Signed-off-by: Oleg Nesterov <oleg@redhat.com>Reviewed-by: Sergey Dyasly <dserrg@gmail.com>Tested-by: Sergey Dyasly <dserrg@gmail.com>Reviewed-by: Sameer Nanda <snanda@chromium.org>Acked-by: David Rientjes <rientjes@google.com>Cc: "Eric W. Biederman" <ebiederm@xmission.com>Cc: Frederic Weisbecker <fweisbec@gmail.com>Cc: Mandeep Singh Baines <msb@chromium.org>Cc: "Ma, Xindong" <xindong.ma@intel.com>Cc: Michal Hocko <mhocko@suse.cz>Cc: "Tu, Xiaobing" <xiaobing.tu@intel.com>Signed-off-by: Andrew Morton <akpm@linux-foundation.org>Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>Signed-off-by: Zefan Li <lizefan@huawei.com>---include/linux/init_task.h | 2 ++include/linux/sched.h | 12 ++++++++++++kernel/exit.c | 1 +kernel/fork.c | 7 +++++++4 files changed, 22 insertions(+)diff --git a/include/linux/init_task.h b/include/linux/init_task.hindex e7bafa4..b11e298 100644--- a/include/linux/init_task.h+++ b/include/linux/init_task.h@@ -38,6 +38,7 @@ extern struct fs_struct init_fs;#define INIT_SIGNALS(sig) { \.nr_threads = 1, \+ .thread_head = LIST_HEAD_INIT(init_task.thread_node), \.wait_chldexit = __WAIT_QUEUE_HEAD_INITIALIZER(sig.wait_chldexit),\.shared_pending = { \.list = LIST_HEAD_INIT(sig.shared_pending.list), \@@ -202,6 +203,7 @@ extern struct task_group root_task_group;[PIDTYPE_SID] = INIT_PID_LINK(PIDTYPE_SID), \}, \.thread_group = LIST_HEAD_INIT(tsk.thread_group), \+ .thread_node = LIST_HEAD_INIT(init_signals.thread_head), \INIT_IDS \INIT_PERF_EVENTS(tsk) \INIT_TRACE_IRQFLAGS \diff --git a/include/linux/sched.h b/include/linux/sched.hindex 56d8233..d529bd9 100644--- a/include/linux/sched.h+++ b/include/linux/sched.h@@ -534,6 +534,7 @@ struct signal_struct {atomic_t sigcnt;atomic_t live;int nr_threads;+ struct list_head thread_head;wait_queue_head_t wait_chldexit; /* for wait4() */@@ -1394,6 +1395,7 @@ struct task_struct {/* PID/PID hash table linkage. */struct pid_link pids[PIDTYPE_MAX];struct list_head thread_group;+ struct list_head thread_node;struct completion *vfork_done; /* for vfork() */int __user *set_child_tid; /* CLONE_CHILD_SETTID */@@ -2397,6 +2399,16 @@ extern bool current_is_single_threaded(void);#define while_each_thread(g, t) \while ((t = next_thread(t)) != g)+#define __for_each_thread(signal, t) \+ list_for_each_entry_rcu(t, &(signal)->thread_head, thread_node)++#define for_each_thread(p, t) \+ __for_each_thread((p)->signal, t)++/* Careful: this is a double loop, 'break' won't work as expected. */+#define for_each_process_thread(p, t) \+ for_each_process(p) for_each_thread(p, t)+static inline int get_nr_threads(struct task_struct *tsk){return tsk->signal->nr_threads;diff --git a/kernel/exit.c b/kernel/exit.cindex 3eb4dcf..38980ea 100644--- a/kernel/exit.c+++ b/kernel/exit.c@@ -74,6 +74,7 @@ static void __unhash_process(struct task_struct *p, bool group_dead)__this_cpu_dec(process_counts);}list_del_rcu(&p->thread_group);+ list_del_rcu(&p->thread_node);}/*diff --git a/kernel/fork.c b/kernel/fork.cindex 0a189c0..be2495f 100644--- a/kernel/fork.c+++ b/kernel/fork.c@@ -1026,6 +1026,11 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk)sig->nr_threads = 1;atomic_set(&sig->live, 1);atomic_set(&sig->sigcnt, 1);++ /* list_add(thread_node, thread_head) without INIT_LIST_HEAD() */+ sig->thread_head = (struct list_head)LIST_HEAD_INIT(tsk->thread_node);+ tsk->thread_node = (struct list_head)LIST_HEAD_INIT(sig->thread_head);+init_waitqueue_head(&sig->wait_chldexit);if (clone_flags & CLONE_NEWPID)sig->flags |= SIGNAL_UNKILLABLE;@@ -1433,6 +1438,8 @@ static struct task_struct *copy_process(unsigned long clone_flags,p->group_leader = current->group_leader;list_add_tail_rcu(&p->thread_group,&p->group_leader->thread_group);+ list_add_tail_rcu(&p->thread_node,+ &p->signal->thread_head);}attach_pid(p, PIDTYPE_PID, pid);nr_threads++;--1.9.1但提交信息没有直白地解释signal_struct->thread_head是什么,不过不要紧,fork()创建子进程时肯定会向子进程的task_struct里面的signal_struct->thread_head设置值的,看提交者修改的代码即可,在copy_signal()中:+ /* list_add(thread_node, thread_head) without INIT_LIST_HEAD() */ + sig->thread_head = (struct list_head)LIST_HEAD_INIT(tsk->thread_node); + tsk->thread_node = (struct list_head)LIST_HEAD_INIT(sig->thread_head);tsk和sig是什么? 这就要从copy_signal()反向推出从fork()开始的调用链了!fork() └─> kernel_clone() └─> copy_process() └─> copy_signal()copy_process()定义在/kernel/fork.c中:__latent_entropy struct task_struct *copy_process( struct pid *pid, int trace, int node, struct kernel_clone_args *args) { struct task_struct *p; //...... p = dup_task_struct(current, node); //复制父进程的task_struct对象,p指向该对象,该函数的细节会在下篇文章说 //..... retval = copy_signal(clone_flags, p); if (retval) goto bad_fork_cleanup_sighand; //...... bad_fork_cleanup_sighand: __cleanup_sighand(p->sighand); //...... }这时再进入copy_signal()看看,定义在/kernel/fork.c中:static int copy_signal(u64 clone_flags, struct task_struct *tsk) { struct signal_struct *sig; if (clone_flags & CLONE_THREAD) return 0; sig = kmem_cache_zalloc(signal_cachep, GFP_KERNEL); tsk->signal = sig; if (!sig) return -ENOMEM; //...... /* list_add(thread_node, thread_head) without INIT_LIST_HEAD() */ sig->thread_head = (struct list_head)LIST_HEAD_INIT(tsk->thread_node); tsk->thread_node = (struct list_head)LIST_HEAD_INIT(sig->thread_head); //...... }给出子进程指针的传递路径:tsk指向的是创建的子进程的task_struct看到kmem_cache_zalloc(),说明sig指向的是刚创建的signal_struct对象的地址,然后把这个地址写入到tsk->signal,这样创建的signal_struct对象就属于这个子进程了接着向signal_struct对象设置值,比如sig->thread_head,设置tsk->thread_node的值:/* list_add(thread_node, thread_head) without INIT_LIST_HEAD() */ sig->thread_head = (struct list_head)LIST_HEAD_INIT(tsk->thread_node); tsk->thread_node = (struct list_head)LIST_HEAD_INIT(sig->thread_head);注: LIST_HEAD_INIT之前在Linux内核的双向链表代码解析(1) 初始化、插入、删除文章讲过展开后:sig->thread_head = (struct list_head){ &(tsk->thread_node), &(tsk->thread_node) } tsk->thread_node = (struct list_head){ &(sig->thread_head), &(sig->thread_head) }那么有:(sig->thread_head).next==&(tsk->thread_node) (sig->thread_head).prev==&(tsk->thread_node) (tsk->thread_node).next==&(sig->thread_head) (tsk->thread_node).prev==&(sig->thread_head)画个图就很容易理解,设置前:设置后,是一个“自循环”的链表:那如果这个刚刚创建子进程创建自己的第一个新线程,上面的图会怎么变化呢?答: 通过pthread_create()创建线程,copy_signal会直接返回,不分配新的signal_struct!!!static int copy_signal(u64 clone_flags, struct task_struct *tsk) { struct signal_struct *sig; if (clone_flags & CLONE_THREAD) //如果是线程,那么不会分配signal_struct,直接返回 return 0; //...... }CLONE_THREAD宏定义在/include/uapi/linux/sched.h:#define CLONE_THREAD 0x00010000 /* Same thread group? */结论: 同一个线程组内,signal_struct只会创建一次那其他线程的task_struct的struct signal_struct* signal指向谁呢?答: 我猜测所有线程共享同一个signal,这里可以调试内核验证,这里为了简单起见,稍微修改一下多线程进程实验的代码://test_thread_head.c #include <pthread.h> #include <stdio.h> #include <unistd.h> void *start_routine(void* arg) //执行这个的叫新线程 { for (;;) { printf("thread %d is running...,PID=%d\n",*(int*)arg,getpid()); sleep(1); } } int main() { pthread_t glibc_tid; int one=1; int ret=pthread_create(&glibc_tid, NULL,start_routine, &one); for (;;) //执行这个的叫主线程 { printf("The process is running... PID=%d,shell's PID=%d\n",getpid(),getppid()); sleep(1); } return 0; }gcc -static test_thread_head.c -o test_thread_head.out再调试内核看看:执行test_thread_head.out,那么shell就fork()+execve()出了test_thread_head.out进程,这个进程再创建一个线程gdb调试时要注意看所有线程的task_struct.signal是不是指向同一个对象,即看看signal指针的值是否一样:发现主线程和新线程输出的的signal值一样,都为0xffff8880044eb180,说明指向的是同一个signal_struct对象!结论: 同一个线程组内,所有线程共享同一个signal_struct对象.体现在所有线程的task_struct.signal的指针的值相同不要退出调试,这时再看看新线程的thread_node的next指向谁:发现next值为0xffff8880044eb190,恰好等于主线程的thread_node对象里面的prev的值(注意,这里说的是两个指针变量本身存储的值是一样的)那0xffff8880044eb190代表什么? 结合之前的画的图,这个值很有可能指向共享的signal_struct的thread_head的成员!使用gdb验证,算出来的值为0xffff8880044eb180,确实是共享的signal_struct对象给出所有的gdb命令和结果,方便读者复制后调试:(gdb) p $lx_task_by_pid(66)->signal $1 = (struct signal_struct *) 0xffff8880044eb180 (gdb) p $lx_task_by_pid(66)->thread_node $2 = {next = 0xffff8880046343a8, prev = 0xffff8880044eb190} (gdb) p ((struct task_struct *)(0xffff8880046343a8 - ((size_t) &((struct task_struct *)0)->thread_node))) $3 = (struct task_struct *) 0xffff888004633d00 (gdb) p ((struct task_struct *)(0xffff888004633d00))->signal $4 = (struct signal_struct *) 0xffff8880044eb180 (gdb) p ((struct task_struct *)(0xffff888004633d00))->thread_node $5 = {next = 0xffff8880044eb190, prev = 0xffff888004637168} (gdb) p ((struct signal_struct *)(0xffff8880044eb190 - ((size_t) &((struct signal_struct *)0)->thread_head))) $6 = (struct signal_struct *) 0xffff8880044eb180可以画创建一个新线程后的图了! 如下:上面的图比较复杂,箭头太多了,可以把上面的图拆开为2个图去理解,如下:图1: signal_struct是所有线程共享的同一个 signal_struct signal_struct (thread_head, nr_threads, ...) ▲ ▲ ▲ ▲ │ │ │ │ signal│ signal│ signal│ signal│ │ │ │ │ 组长.task 新线程1.task 新线程2.task ... 最后线程.task thread_node thread_node thread_node thread_node图2: 线程组内所有线程之间的组织情况附memraid代码,然后放到Free Mermaid Editor Online | PNG, SVG & PDF | Mermaid Live Editor里面生图flowchart TD subgraph SIG["signal_struct"] TH["signal->thread_head (哨兵)"] end subgraph TS0["task_struct (主线程/组长)"] T0["主线程 (组长).thread_node"] end subgraph TS1["task_struct (新线程1)"] T1["新线程1.thread_node"] end DOTS["..."] subgraph TSn["task_struct (新线程n)"] Tn["新线程n.thread_node"] end TH <--> T0 T0 <--> T1 T1 <-->DOTS DOTS<-->Tn Tn <--> TH结论同一个线程组内,所有线程的 task_struct.signal都共享同一个signal_struct对象,该对象是主线程创建时分配空间的,主线程创建的线程不分配signal_struct对象thread_head.next指向主线程的thread_node,thread_head.prev指向最后一个线程的 thread_node,通过thread_head就能遍历所有同一个线程组内的所有线程